task_wdt: Task watchdog got triggered ESP32S3 AsyncWebServer

[kekykeke]

Hello,

I dealing with a similar issue as described here:

https://github.com/espressif/arduino-esp32/issues/2267

I wonder how this issue never been solved ... None of the answers are satisfactory in the thread.



In this guy's case something was holding up the execution, in my case I have a long delay, about 50 secons in my code upon a command while the ESP performs a sequence, it turns on some machine, keeps monitoring the status then goes back to the main loop.

I could put a hold timer of course on this to spend as least time it's possible inside this function but as I have said this is a sequence, the Arduino or in this case the ESP waiting back some input from the machine and proceeds further or it fails.

I really don't want to rewrite my code just for this and I don't get it either, this thing 20 folds more powerful as an Arduino and has 2 CPUs, why can't the other core just do the webstuff or if it can then HOW?

My plan is that once the sequence has been started I set a global variable so if someone reloads the page or visits the same website he just sees that the machine is busy, once this flag cleared out the website would show the turn on button again.

Code: Select all

14:32:04.450 -> E (79213) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
14:32:04.450 -> E (79213) task_wdt:  - async_tcp (CPU 0/1)
14:32:04.450 -> E (79213) task_wdt: Tasks currently running:
14:32:04.450 -> E (79213) task_wdt: CPU 0: IDLE
14:32:04.450 -> E (79213) task_wdt: CPU 1: loopTask
14:32:04.450 -> E (79213) task_wdt: Aborting.
14:32:04.483 -> 
14:32:04.483 -> abort() was called at PC 0x42017144 on core 0
14:32:04.483 -> 
14:32:04.483 -> 
14:32:04.483 -> Backtrace: 0x4037718e:0x3fc95560 0x4037c949:0x3fc95580 0x4038269d:0x3fc955a0 0x42017144:0x3fc95620 0x40378685:0x3fc95640 0x420843c7:0x3fcf47d0 0x420179a6:0x3fcf47f0 0x4037dedc:0x3fcf4810
14:32:04.483 -> 
14:32:04.483 -> 
14:32:04.483 -> 
14:32:04.483 -> 
14:32:04.483 -> ELF file SHA256: 7e29832554f34c22
14:32:04.649 -> 
14:32:04.649 -> Rebooting...
14:32:04.649 -> ESP-ROM:esp32s3-20210327
14:32:04.649 -> Build:Mar 27 2021
14:32:04.649 -> rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
14:32:04.649 -> Saved PC:0x40376d9c
14:32:04.649 -> SPIWP:0xee
14:32:04.649 -> mode:DIO, clock div:1
14:32:04.649 -> load:0x3fce3808,len:0x44c
14:32:04.649 -> load:0x403c9700,len:0xbe4
14:32:04.649 -> load:0x403cc700,len:0x2a68
14:32:04.649 -> entry 0x403c98d4

[MicroController]

in my case I have a long delay, about 50 secons in my code upon a command while the ESP performs a sequence

Do not run this code in the HTTP server's task/callback. Let the processing have it own task.

this thing 20 folds more powerful as an Arduino and has 2 CPUs, why can't the other core just do the webstuff or if it can then HOW?

Of course it can. Easily. But you must provide different tasks which can then run concurrently.

[kekykeke]

I have found a "solution", not really what I wanted but this guy explained this quite well:

he way the question is formulated, I assume you don't know what a watchdog is nor why it is triggering. Hence:


Arduino for the ESP is built on the ESP-IDF, which in turn is built around FreeRTOS. FreeRTOS creates one IDLE task per core. It also sets watchdog timers on these Tasks. This means if these tasks get starved of execution time, then after a timeout period, the watchdog gets triggered and resets the chip. The IDLE tasks do some important FreeRTOS "household" work in the background, so you have to give them time. They also have the lowest possible priority. So if any task is running with a higher priority (such as your callbacks), these will always run first while the IDLE tasks will have to wait. Thus all higher priority tasks must be short enough to avoid triggering the watchdog. If this isn't feasible, you have to insert pauses in sufficient intervals, for example by calling vTaskDelay(...) or by performing some IO function that blocks long enough. In either case, the current task will go to sleep, and FreeRTOS will kick in another task, and if there are no other higher priority tasks waiting, it will finally allow the IDLE tasks to execute. What all this means is that usually no code you write should ever preoccupy the CPU for 100% of the time for any period longer than the watchdog timeout period. Moreover, you can't turn this off in Arduino. It is only configurable in ESP-IDF, meaning you have to code within that framework or recompile Arduino on your own. Moreover it is a bad idea to turn it off anyways.


You must use a value of atleast 1 in the call to vTaskDelay(...). Due to integer arithmetic and the various expressions passed as a parameter, it could end up amounting to 0, ie no delay at all.

If you really need to run a task without pause, you'll have to create the task on your own and give it a priority of tskIDLE_PRIORITY, such as:


xTaskCreate(someFunction, "HumanReadableNameofTask", 4096, NULL, tskIDLE_PRIORITY, NULL);

where your function has the following signature:

void someFunction(void* arg) {...}

You can do this in Arduino too. So, either keep the code that runs in the callbacks to a minimum, or shift the heavy lifting to a separate task and have the callbacks simply forward any relevant information to that task (such as using volatile variables and semaphores, ie the usual methods of synchronization in parallel processing. that's beyond the scope of this answer).

Note I'm not sure if calling delay(...) in Arduino has the same effect as vTaskDelay, but it probably does (with some subtleties). Also, there is an official way for a task to "yield" to another lower task, I'm not sure about the details though.

Important: Using delays (such as vTaskDelay(...)) within callbacks, especially timer callbacks, is a bad idea, since they block other callbacks (within the same task) from executing. Hence the best option is to pass information to a separate task running at idle priority (ie priority 0). The reason why a task running at 0 priority doesn't trigger the watchdog, is because the IDLE tasks also have that priority, and FreeRTOS round-robins all those tasks at the same priority, ie gives them interleaving slices of time to execute in "parallel". But when you have two tasks with differing priorites, the higher one ALWAYS executes until it is done or stalls/sleeps, only then will the lower priority run, either until it itself finishes, or until the higher priority task wakes up again and demands execution time.

Update:

Keeping the Arduino loop() empty, will definately trigger the wdt, because it isn't really "empty", because the loop() function is internally wrapped in an infinite loop, so the CPU consumption would actually shoot to 100% without doing anything useful. That is what is triggering the watchdog. The GET process seems to be happening asynchronously, so it is not blocking the loop() from executing.

I tried to move everything from that function which has the big delay into a thread and call the taskcreate from the original function but it crashes with some corruption error :/

[kekykeke]

I start understanding now after some examples that the taskcreates should be in the setup() section and not later on in the program.

The following little codesnippet works however I added my modification this way:

Code: Select all

#include <Arduino.h>

TaskHandle_t TaskHandle_1;
TaskHandle_t TaskHandle_2;

int count1 = 0;
int count2 = 0;

/* Task1 with priority 1 */
static void MyTask1(void* pvParameters)
{
  while(1)
  {
  if(start_process)
  {
    my_long_function();
    Serial.println(F("Task1 counter: "));
    Serial.println(count1++);
    }
    
    vTaskDelay(1000 / portTICK_PERIOD_MS);
 //   vTaskDelete(TaskHandle_1);    // Delete the task using the TaskHandle_1
  }
}

/* Task2 with priority 2 */
static void MyTask2(void* pvParameters)
{
  while(1)
  {    
    Serial.println(F("Task2 counter: "));
    Serial.println(count2++);
    vTaskDelay(1000 / portTICK_PERIOD_MS);
//    vTaskDelete(NULL);     //Delete own task by passing NULL(TaskHandle_2 can also be used)
  }
}


void setup()
{
  
  Serial.begin(9600);
  Serial.println(F("In Setup function"));

  /* Create two tasks with priorities 1 and 3. Capture the Task details to respective handlers */
  xTaskCreate(MyTask1, "Task1", 1000, NULL, 1, &TaskHandle_1);  
  xTaskCreate(MyTask2, "Task2", 1000, NULL, 3, &TaskHandle_2);

}

void loop()
{
  // Hooked to IDle task, it will run whenever CPU is idle
  Serial.println(F("Loop function"));
  delay(50);
}

Now it is true that the my_long_function() does not interfere with the webserver process anymore and it's even run however after finished running the ESP32 crashes with:

Code: Select all

Guru Meditation Error: Core  0 panic'ed (Unhandled debug exception). 
Debug exception reason: Stack canary watchpoint triggered (Task1) 
Core  0 register dump:

Core  0 register dump:
PC      : 0x4037fb57  PS      : 0x00060e36  A0      : 0x803822b4  A1      : 0x3fcec1a0
A2      : 0x3fc9ff38  A3      : 0xb33fffff  A4      : 0x0000abab  A5      : 0x00060e23
A6      : 0x00060e20  A7      : 0x0000cdcd  A8      : 0xb33fffff  A9      : 0xffffffff
A10     : 0x3fcec338  A11     : 0x3fcf19f8  A12     : 0x3c091fef  A13     : 0x0000001b
A14     : 0x02c9ff38  A15     : 0x00ffffff  SAR     : 0x00000008  EXCCAUSE: 0x00000001
EXCVADDR: 0x00000000  LBEG    : 0x400556d5  LEND    : 0x400556e5  LCOUNT  : 0xfffffffe


Backtrace: 0x4037fb54:0x3fcec1a0 0x403822b1:0x3fcec1e0 0x4038240d:0x3fcec200 0x403776b3:0x3fcec220 0x403776c5:0x3fcec250 0x403776ee:0x3fcec270 0x403827b5:0x3fcec290 0x4200984f:0x3fcec2b0 0x42008e8d:0x3fcec330 0x42004a63:0x3fcec360 0x42004bbb:0x3fcec3d0 0x42004cf4:0x3fcec410

What am I doing wrong?

Further more: would it have any advantage of running the Async Webserver as a separate task not in the main loop?

Thanks

[ESP_Sprite]

Your task1 and task2 only have 1000 bytes of RAM, and that's too few, leading to that specific error (Stack canary watchpoint triggered). Give them more stack (8192 or more) and the error probably goes away.

[kekykeke]

Yes you are correct but what is this used for? Is this memory some sort of storage area of the commands inside the thread?
Will it get automatically cleaned up? As you see I have while loop inside the task so that will be going on forever, when the global variable unlocks then it executes the long function then returns into that while loop, is this any problem?

Can you confirm this: unless delay actually calls vTaskDelay on esp32, which it does. delay does not block on esp32!

So basically all the delay() 's in my code are equal to the vTaskDelay and won't interfere with anything.

Thanks

[MicroController]

what is this used for? Is this memory some sort of storage area of the commands inside the thread?
Will it get automatically cleaned up?

Every thread/task needs its own memory for its stack. This is basically where all variables which are declared inside functions ("local variables", i.e. not "global variables" and not "heap memory") live. The compiler automatically takes care of putting variables into this stack when a function is called and removing them again when the function returns.
When you create a task you have to tell the OS how much memory it should reserve for this task's stack. When a task is deleted, the OS also releases this memory again.
What amount of stack space a task requires is very hard to tell in advance. However, as some guideline: The minimum permissible stack size in ESP-IDF by default is about 1.5kb, the default stack size is about 3.5kb. Some tasks need more than that.
Too little stack space will cause crashes/errors/... as you have witnessed. Too much stack space may waste memory.

As you see I have while loop inside the task so that will be going on forever, when the global variable unlocks then it executes the long function then returns into that while loop, is this any problem?

No problem at all. This is how most applications work.