How to debug ESP32interrupt watchdog timeout

[yake827]

Hi ,
I've been struggling with an ESP32 interrupt watchdog issue recently. I've been reading the introduction to interrupts(https://docs.espressif.com/projects/esp ... t-watchdog) in ESP-IDF, but it still doesn't solve my problem.

For a normal crash problem, we can infer the crash by backtrace, but this seems does not apply to the interrupt watchdog. In fact, I can't understand why the backtrace has no reference when it comes to triggering the interrupt watchdog. According to the documentation, when an interrupt watchdog timeout, the interrupt has been disabled and the FreeRTOS schedule should be stopped, why the backtrace points to another blocked task .

[ESP_igrr]

Hi yake827, can you please post the backtrace you are seeing?

Interrupts being disabled is one possible cause of interrupt watchdog timeout. Another possible cause is that an interrupt handler didn't complete in time (entered an infinite loop) or that an interrupt handler keeps re-entering (some interrupt status bit wasn't cleared).

[yake827]

Hi igrr,
Thanks for your reply, The crash backtrace is shown below.

Code: Select all

[2021-07-20 11:35:39 337]Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0)
[2021-07-20 11:35:39 337]Core 0 register dump:
[2021-07-20 11:35:39 337]PC      : 0x400915d2  PS      : 0x00060b34  A0      : 0x800903ed  A1      : 0x3ffd4e80  
[2021-07-20 11:35:39 337]A2      : 0x3ffd3598  A3      : 0x3ffd4ff0  A4      : 0x00000c9f  A5      : 0x3f408990  
[2021-07-20 11:35:39 337]A6      : 0x3ffbf580  A7      : 0x00060023  A8      : 0x3ffd4ff0  A9      : 0x0000000f  
[2021-07-20 11:35:39 338]A10     : 0x3ffd4ff0  A11     : 0x0000000f  A12     : 0x00060b20  A13     : 0x00000001  
[2021-07-20 11:35:39 373]A14     : 0x3ffeddf0  A15     : 0x3ffea57c  SAR     : 0x00000018  EXCCAUSE: 0x00000005  
[2021-07-20 11:35:39 373]EXCVADDR: 0x00000000  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xfffffffe  
[2021-07-20 11:35:39 373]
[2021-07-20 11:35:39 373]ELF file SHA256: eadf38922e335ed1
[2021-07-20 11:35:39 373]
[2021-07-20 11:35:39 373]Backtrace: 0x400915cf:0x3ffd4e80 0x400903ea:0x3ffd4ea0 0x4008f514:0x3ffd4ec0 0x400ffb91:0x3ffd4f00 0x40136612:0x3ffd4f30

I used the add2line print the backtrace.

Code: Select all

0x400915cf
vListInsert
/arm/esp-idf/components/freertos/list.c:205
0x400903ea
vTaskPlaceOnEventList
/arm/esp-idf/components/freertos/tasks.c:2901 (discriminator 2)
0x4008f514
xQueueGenericReceive
/arm/esp-idf/components/freertos/queue.c:1596
0x400ffb91
arch_os_mbox_fetch
/arm/miio_project/components/miio_sdk/arch/esp32/arch_os.c:352 (discriminator 4)
0x40136612
msg_fetch_block
/arm/miio_project/components/miio_sdk/miio_common/libs/d0/delayzero.c:631

Using backtrace, I traced the code to the following.

code1.png (48.63 KiB) Viewed 6334 times

This code is just a loop waiting for the data, which is triggered by the mbedTLS to read the data.

So my question is why backtrace prints these information. If according to your condition of triggering the interrupt watchdog, I don't think it should print this task, because context switch has been prohibited.

[MatheusGSilva]

Can't answer about the backtrace, but the problem is probably the xQueueReceiveFromISR. This function is meant to receive data inside the ISR, and not from one ISR.

Best regards.

[ESP_Sprite]

Can't answer about the backtrace, but the problem is probably the xQueueReceiveFromISR. This function is meant to receive data inside the ISR, and not from one ISR.

Disagree, that bit of code is only called if the entire function is called form an ISR (portIsInISR). Additionally, I don't think there's anything wrong with calling *FromISR functions in a non-ISR situation, you just don't get to do any blocking.

[yake827]

arch_os_mbox_fetch function is not called inside an interrupt callback, which is where I get confused.
In my understanding, if not disable interrupt, then the watchdog must be in the ISR function infinite loop when triggered, should be obvious from the backtrace.

[ESP_Sprite]

To be honest, it's hard to guess from the backtrace alone, but it smells like memory corruption, or something like use-after-free... the top function called is vListInsert, which tends to run in a critical region. If the list that needs inserting into is somehow corrupted, that function could possibly loop forever trying to find the insertion point. It's far from a smoking gun here, but you could see what happens if you enable the various memory debugging options in ESP-IDF.

[yake827]

Hi Sprite, thank you very much for your thoughts. You remind me that this may just seem to be an interrupt watchdog issue. I will rearrange my orientation.