Help with Interrupt WDT Timeout Resets

[jwwest]

My project is experiencing intermittent resets due to the Interrupt WDT Timeout. I have a number of Guru meditation register dumps and backtraces as well as the corresponding core-dumps. Below I have inserted copies of a guru dump and the core dump from a single crash occurrence.

These crashes consistently have calls to the i2c_isr_handler_default in current thread and Core 0 always seems to be in ISR context.
There are 2 ISR functions that get printed, uart_hal_write_txfifo and i2c_isr_handler_default... but the EPCx registers in the guru dump don't exactly match what the core dump produced.

IDF version is 4.2.2.

Ultimately what I would like answered is how to make sense of the guru dump and the core dump... understanding the meaning of the different registers, especially the Exception registers, and the call stacks. Also, how is the stack managed when in ISR context?

The core dump has some issues, which I suspect may be due to heap corruption, or maybe stack overflow, but I've increased the stack sizes and still experience the problem as similar rates. There are 2 threads that tend to get stomped on in the core dump, and the threads info section lists a bunch of "New process nnnnn" messages at the end, which I don't understand.


Guru dump...

Code: [Select all] [Expand/Collapse]

1. Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0).
2.  
3. Core  0 register dump:
4. PC      : 0x400848e3  PS      : 0x00050434  A0      : 0x400826c8  A1      : 0x3ffb19d0
5. 0x400848e3: i2c_isr_handler_default at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/i2c.c:463
6.  
7. 0x400826c8: _xt_lowint1 at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/xtensa/xtensa_vectors.S:1105
8.  
9. A2      : 0x3ffc4e6c  A3      : 0x00000000  A4      : 0x00000000  A5      : 0x00000000
10. A6      : 0x3ffb67f8  A7      : 0x00000000  A8      : 0x00000000  A9      : 0x3ffb19a0
11. A10     : 0x3ffc4f54  A11     : 0x00060421  A12     : 0x3ffb19d8  A13     : 0x00000000
12. A14     : 0x3ffc4f54  A15     : 0x3ffc4fa8  SAR     : 0x0000001e  EXCCAUSE: 0x00000005
13. EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff
14. Core  0 was running in ISR context:
15. EPC1    : 0x401982eb  EPC2    : 0x4000bff0  EPC3    : 0x00000000  EPC4    : 0x400848e3
16. 0x401982eb: uart_hal_write_txfifo at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/hal/uart_hal_iram.c:35
17.  
18. 0x400848e3: i2c_isr_handler_default at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/i2c.c:463
19.  
20.  
21. Backtrace:0x400848e0:0x3ffb19d0 0x400826c5:0x3ffb1a00 0x400848f3:0x3ffd0770 0x4008ced5:0x3ffd0780 0x4008c583:0x3ffd07a0 0x400f67c0:0x3ffd07c0
22. 0x400848e0: i2c_isr_handler_default at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/i2c.c:460
23.  
24. 0x400826c5: _xt_lowint1 at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/xtensa/xtensa_vectors.S:1105
25.  
26. 0x400848f3: i2c_isr_handler_default at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/i2c.c:488
27.  
28. 0x4008ced5: vPortExitCritical at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/xtensa/port.c:457
29.  
30. 0x4008c583: ulTaskNotifyTake at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:4651
31.  
32. 0x400f67c0: zeroCrossingTask at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/triac-driver/zero-crossing.c:92 (discriminator 1)
33.  
34. Core  1 register dump:
35. PC      : 0x4008cec1  PS      : 0x00060a34  A0      : 0x8008dce6  A1      : 0x3ffcb2e0
36. 0x4008cec1: vPortExitCritical at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/xtensa/port.c:448
37.  
38. A2      : 0x3ffb67fc  A3      : 0x3ffcb350  A4      : 0x000005b6  A5      : 0x3f437df0
39. A6      : 0x00000000  A7      : 0x3ffc9d80  A8      : 0x00000004  A9      : 0x00060a23
40. A10     : 0x0000abab  A11     : 0x3ffc4fb4  A12     : 0x00000004  A13     : 0x3ffcb354
41. A14     : 0x00000000  A15     : 0x00000000  SAR     : 0x00000019  EXCCAUSE: 0x00000005
42. EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff
43.  
44. Backtrace:0x4008cebe:0x3ffcb2e0 0x4008dce3:0x3ffcb300 0x40109afb:0x3ffcb340 0x40143b96:0x3ffcb380 0x4014362a:0x3ffcb3a0 0x40143a6e:0x3ffcb3e0 0x40100179:0x3ffcb400 0x400fff39:0x3ffcb420 0x400ffab7:0x3ffcb440
45. 0x4008cebe: vPortExitCritical at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/xtensa/port.c:448
46.  
47. 0x4008dce3: xQueueGenericReceive at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/queue.c:1556
48.  
49. 0x40109afb: i2c_master_cmd_begin at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/i2c.c:1176
50.  
51. 0x40143b96: oven_i2c_write at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/led-driver/oven-i2c.c:117
52.  
53. 0x4014362a: writeWholeBuffer at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/led-driver/ledDriver.c:146
54.  (inlined by) writeWholeBuffer at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/led-driver/ledDriver.c:129
55.  
56. 0x40143a6e: drawLedFlush at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/led-driver/ledDriver.c:434 (discriminator 1)
57.  
58. 0x40100179: __ui_instance_flush at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/UI/ui_instance_hw1.c:366
59.  
60. 0x400fff39: __ui_instance_wrapper_flush at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/UI/ui_instance_core.c:211
61.  
62. 0x400ffab7: uiThread at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/UI/ui.c:1915
63.  
64.  
65.  
66. ELF file SHA256: b904f1e13b0fb91b

GeSHi © Codebox Plus Extension

Coredump...

Code: [Select all] [Expand/Collapse]

1.  
2. ===============================================================
3. ==================== ESP32 CORE DUMP START ====================
4.  
5. Crashed task handle: 0x3ffd0884, name: 'zeroCrossingTas', GDB name: 'process 1073547396'
6.  
7. ================== CURRENT THREAD REGISTERS ===================
8. exccause       0x5 (AllocaCause)
9. excvaddr       0x0
10. epc1           0x4000bff0
11. epc2           0x0
12. epc3           0x400848e3
13. epc4           0x0
14. epc5           0x0
15. epc6           0x0
16. epc7           0x0
17. eps2           0x60421
18. eps3           0x0
19. eps4           0x50421
20. eps5           0x0
21. eps6           0x0
22. eps7           0x40089aae
23. pc             0x400848e3   0x400848e3 <i2c_isr_handler_default+143>
24. lbeg           0x4000c2e0   1073791712
25. lend           0x4000c2f6   1073791734
26. lcount         0xffffffff   4294967295
27. sar            0x1e 30
28. ps             0x50424  328740
29. threadptr      <unavailable>
30. br             <unavailable>
31. scompare1      <unavailable>
32. acclo          <unavailable>
33. acchi          <unavailable>
34. m0             <unavailable>
35. m1             <unavailable>
36. m2             <unavailable>
37. m3             <unavailable>
38. expstate       <unavailable>
39. f64r_lo        <unavailable>
40. f64r_hi        <unavailable>
41. f64s           <unavailable>
42. fcr            <unavailable>
43. fsr            <unavailable>
44. a0             0x400826c8   1074276040
45. a1             0x3ffb19d0   1073420752
46. a2             0x3ffc4e6c   1073499756
47. a3             0x0  0
48. a4             0x0  0
49. a5             0x0  0
50. a6             0x3ffb67f8   1073440760
51. a7             0x0  0
52. a8             0x0  0
53. a9             0x3ffb19a0   1073420704
54. a10            0x3ffc4f54   1073499988
55. a11            0x60421  394273
56. a12            0x3ffb19d8   1073420760
57. a13            0x0  0
58. a14            0x3ffc4f54   1073499988
59. a15            0x3ffc4fa8   1073500072
60.  
61. ==================== CURRENT THREAD STACK =====================
62. #0  0x400848e3 in i2c_isr_handler_default (arg=0x3ffc4e6c) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/i2c.c:460
63. #1  0x400826c8 in _xt_lowint1 () at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/xtensa/xtensa_vectors.S:1105
64. #2  0x400848f6 in i2c_isr_handler_default (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/i2c.c:488
65. #3  0x4008ced8 in vPortExitCritical (mux=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/xtensa/port.c:457
66. #4  0x4008c586 in ulTaskNotifyTake (xClearCountOnExit=1, xTicksToWait=4294967295) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:4651
67. #5  0x400f67c3 in zeroCrossingTask (pvParameters=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/triac-driver/zero-crossing.c:92
68.  
69. ======================== THREADS INFO =========================
70.   Id   Target Id         Frame
71. * 1    process 1073547396 0x400848e3 in i2c_isr_handler_default (arg=0x3ffc4e6c) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/i2c.c:460
72.   2    process 1073532180 0x4000bff0 in _xtos_set_intlevel ()
73.   3    process 1073549588 0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffd1290, pvBuffer=0x3ffd1050, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
74.   4    process 1073526148 0x00000000 in ?? ()
75.   5    process 1073610168 spin_task (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/oven/main/stats.c:142
76.   6    process 1073609788 0x400d5552 in spin_task (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/oven/main/stats.c:141
77.   7    process 1073479616 0x40193956 in esp_pm_impl_waiti () at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp32/pm_esp32.c:501
78.   8    process 1073481552 0x40193956 in esp_pm_impl_waiti () at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp32/pm_esp32.c:501
79.   9    process 1073509092 vTaskDelay (xTicksToDelay=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:1489
80.   10   process 1073545716 vTaskDelayUntil (pxPreviousWakeTime=<optimized out>, xTimeIncrement=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:1423
81.   11   process 1073592948 0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffdabf0, pvBuffer=0x0, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
82.   12   process 1073473980 0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffbeb80, pvBuffer=0x0, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
83.   13   process 1073598780 vTaskDelay (xTicksToDelay=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:1489
84.   14   process 1073485324 0x4008cbb3 in prvProcessTimerOrBlockTask (xListWasEmpty=<optimized out>, xNextExpireTime=994448) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
85.   15   process 1073609296 0x4008dd41 in xQueueGenericReceive (/builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/gdb/gdb/inline-frame.c:167: internal-error: void inline_frame_this_id(frame_info*, void**, frame_id*): Assertion `frame_id_p (*this_id)' failed.
86. A problem internal to GDB has been detected,
87. further debugging may prove unreliable.
88. Quit this debugging session? (y or n) [answered Y; input not from terminal]
89. /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/gdb/gdb/inline-frame.c:167: internal-error: void inline_frame_this_id(frame_info*, void**, frame_id*): Assertion `frame_id_p (*this_id)' failed.
90. A problem internal to GDB has been detected,
91. further debugging may prove unreliable.
92. Create a core file of GDB? (y or n) [answered Y; input not from terminal]
93. ERROR: GDB exited (None / )!
94. ERROR: Problem occured! GDB exited, restart it.
95. Reading symbols from build/oven-controller.elf...done.
96. [New process 1073547396]
97. [New process 1073532180]
98. [New process 1073549588]
99. [New process 1073526148]
100. [New process 1073610168]
101. [New process 1073609788]
102. [New process 1073479616]
103. [New process 1073481552]
104. [New process 1073509092]
105. [New process 1073545716]
106. [New process 1073592948]
107. [New process 1073473980]
108. [New process 1073598780]
109. [New process 1073485324]
110. [New process 1073609296]
111. [New process 1073555680]
112. [New process 1073519604]
113. [New process 1073617060]
114. [New process 1073469452]
115. [New process 1073471212]
116. [New process 1073566312]
117. [New process 1073412956]
118. [New process 1073505624]
119. [New process 1073488772]
120. [Current thread is 1 (process 1073547396)]
121.  
122. ==================== THREAD 1 (TCB: 0x3ffd0884, name: 'zeroCrossingTas') =====================
123. #0  0x400848e3 in i2c_isr_handler_default (arg=0x3ffc4e6c) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/i2c.c:460
124. #1  0x400826c8 in _xt_lowint1 () at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/xtensa/xtensa_vectors.S:1105
125. #2  0x400848f6 in i2c_isr_handler_default (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/i2c.c:488
126. #3  0x4008ced8 in vPortExitCritical (mux=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/xtensa/port.c:457
127. #4  0x4008c586 in ulTaskNotifyTake (xClearCountOnExit=1, xTicksToWait=4294967295) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:4651
128. #5  0x400f67c3 in zeroCrossingTask (pvParameters=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/triac-driver/zero-crossing.c:92
129.  
130. ==================== THREAD 2 (TCB: 0x3ffccd14, name: 'ocbThread') =====================
131. #0  0x4000bff0 in ?? ()
132. #1  0x4008beb6 in xTaskGetSchedulerState () at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:4019
133. #2  0x4008d99c in xQueueGenericSend (xQueue=0x3ffcdd3c, pvItemToQueue=0x0, xTicksToWait=<optimized out>, xCopyPosition=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/queue.c:738
134. #3  0x4010f288 in uart_tx_all (uart_num=<optimized out>, src=<optimized out>, size=<optimized out>, brk_en=<optimized out>, brk_len=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/uart.c:1112
135. #4  0x4010f7a8 in uart_tx_all (brk_len=0, brk_en=false, size=1, src=0x3ffcc5c0 \"[\", uart_num=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/uart.c:1121
136. #5  uart_write_bytes (uart_num=0, src=0x3ffcc5c0 \"[\", size=1) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/uart.c:1121
137. #6  0x400f6f2e in uart_tx_char_via_driver (fd=0, c=91) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/vfs/vfs_uart.c:177
138. #7  0x400f7150 in uart_write (fd=0, data=<optimized out>, size=128) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/vfs/vfs_uart.c:221
139. #8  0x400f7810 in esp_vfs_write (r=<optimized out>, fd=1, data=0x3ffbe568, size=128) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/vfs/vfs.c:420
140. #9  0x4000bd86 in ?? ()
141. #10 0x40001180 in ?? ()
142. #11 0x40059301 in ?? ()
143. #12 0x4005937d in ?? ()
144. #13 0x40058b6c in ?? ()
145. #14 0x400da496 in __sprint_r (uio=0x3ffcc944, fp=0x3ffae98c, ptr=0x3ffccd80) at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/vfprintf.c:433
146. #15 __sprint_r (ptr=0x3ffccd80, fp=0x3ffae98c, uio=0x3ffcc944) at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/vfprintf.c:403
147. #16 0x400dfa62 in _vfprintf_r (data=<optimized out>, fp=<optimized out>, fmt0=<optimized out>, ap=...) at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/vfprintf.c:1781
148. #17 0x400dfbd8 in vprintf (fmt=0x3f406ff0 \"  ---  \", ap=...) at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/vprintf.c:34
149. #18 0x400f4e64 in tcpLogPrint (msg=0x3f406ff0 \"  ---  \", args=...) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/tcp/tcp.c:220
150. #19 0x400f4f00 in tcpPrintfRedef (format=0x3f406ff0 \"  ---  \") at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/tcp/tcp.c:49
151. #20 0x400eb4db in ovenMainCook (msg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/oven-control-block/oven-control.c:1751
152. #21 0x400eaae0 in ocbThread (pvParameters=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/oven-control-block/oven-control.c:444
153.  
154. ==================== THREAD 3 (TCB: 0x3ffd1114, name: 'powerConsumptio') =====================
155. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffd1290, pvBuffer=0x3ffd1050, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
156. #1  0x400f63b0 in powerConsumption (pvParameters=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/triac-driver/triac.c:557
157.  
158. ERROR: GDB/MI command failed (error / msg="Cannot access memory at address 0x60a17")!
159. WARNING: Unable to switch to thread 4
160.  
161. ==================== THREAD 5 (TCB: 0x3ffdfdb8, name: 'spin1') =====================
162. #0  spin_task (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/oven/main/stats.c:142
163.  
164. ==================== THREAD 6 (TCB: 0x3ffdfc3c, name: 'spin0') =====================
165. #0  0x400d5552 in spin_task (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/oven/main/stats.c:141
166.  
167. ==================== THREAD 7 (TCB: 0x3ffbffc0, name: 'IDLE0') =====================
168. #0  0x40193956 in esp_pm_impl_waiti () at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp32/pm_esp32.c:501
169. #1  0x400d4b1c in esp_vApplicationIdleHook () at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_common/src/freertos_hooks.c:63
170. #2  0x4008ad77 in prvIdleTask (pvParameters=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:3386
171.  
172. ==================== THREAD 8 (TCB: 0x3ffc0750, name: 'IDLE1') =====================
173. #0  0x40193956 in esp_pm_impl_waiti () at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp32/pm_esp32.c:501
174. #1  0x400d4b1c in esp_vApplicationIdleHook () at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_common/src/freertos_hooks.c:63
175. #2  0x4008ad77 in prvIdleTask (pvParameters=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:3386
176.  
177. ==================== THREAD 9 (TCB: 0x3ffc72e4, name: 'SDM Main Task') =====================
178. #0  vTaskDelay (xTicksToDelay=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:1489
179. #1  0x400f347d in sdm_acquire_temp_afe_one (channel=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/sensor-data-manager/sdm_main.c:1501
180. #2  sdm_read_TEMPn (TEMPn=<optimized out>, slot=<optimized out>, slot_max=5) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/sensor-data-manager/sdm_main.c:1595
181. #3  0x400f35bf in sdm_main_task (pvParameters=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/sensor-data-manager/sdm_main.c:415
182.  
183. ==================== THREAD 10 (TCB: 0x3ffd01f4, name: 'sensorMonitorTh') =====================
184. #0  vTaskDelayUntil (pxPreviousWakeTime=<optimized out>, xTimeIncrement=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:1423
185. #1  0x40140be2 in RunOnInterval (task=0x400f3b90 <monitors>, milliSeconds=<optimized out>, args=0x3ffb6944 <threadArgs$11511>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/helpers/helpers.c:17
186. #2  0x400f3f06 in sensorMonitorThread (arg=0x3ffb6944 <threadArgs$11511>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/sensor-monitor/sensor-monitor-thread.c:261
187.  
188. ==================== THREAD 11 (TCB: 0x3ffdba74, name: 'touchSvc') =====================
189. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffdabf0, pvBuffer=0x0, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
190. #1  0x4010295e in touch_pads_reader_task (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/UI/touch_pad_service.c:424
191.  
192. ==================== THREAD 12 (TCB: 0x3ffbe9bc, name: 'websocket_task') =====================
193. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffbeb80, pvBuffer=0x0, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
194. #1  0x40157591 in sys_arch_sem_wait (sem=0x3ffbeb68, timeout=1000) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/lwip/port/esp32/freertos/sys_arch.c:184
195. #2  0x40146809 in lwip_select (maxfdp1=55, readset=0x3ffe3898, writeset=0x0, exceptset=0x3ffe3890, timeout=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/lwip/lwip/src/api/sockets.c:2153
196. #3  0x400f7e70 in esp_vfs_select (nfds=55, readfds=0x3ffe3898, writefds=0x0, errorfds=0x3ffe3890, timeout=0x3ffe38a0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/vfs/vfs.c:985
197. #4  0x401843f5 in base_poll_read (t=0x3ffbf048, timeout_ms=1000) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/tcp_transport/transport_ssl.c:150
198. #5  0x40195caa in esp_transport_poll_read (t=<optimized out>, timeout_ms=1000) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/tcp_transport/transport.c:156
199. #6  0x401923aa in ws_poll_read (t=0x3ffbf12c, timeout_ms=1000) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/tcp_transport/transport_ws.c:528
200. #7  0x40195caa in esp_transport_poll_read (t=<optimized out>, timeout_ms=1000) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/tcp_transport/transport.c:156
201. #8  0x4018f168 in esp_websocket_client_task (pv=0x3ffd568c) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_websocket_client/esp_websocket_client.c:689
202.  
203. ==================== THREAD 13 (TCB: 0x3ffdd13c, name: 'otaCheckTask') =====================
204. #0  vTaskDelay (xTicksToDelay=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/tasks.c:1489
205. #1  0x400e974e in otaCheckTask (pvParameter=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/ota/ota.c:766
206.  
207. ==================== THREAD 14 (TCB: 0x3ffc160c, name: 'Tmr Svc') =====================
208. #0  0x4008cbb3 in prvProcessTimerOrBlockTask (xListWasEmpty=<optimized out>, xNextExpireTime=994448) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
209. #1  prvTimerTask (pvParameters=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/freertos/timers.c:544
210.  
211. ERROR: GDB exited (None / )!
212. ERROR: Problem occured! GDB exited, restart it.
213. ERROR: GDB exited (None / )!
214. ERROR: Problem occured! GDB exited, restart it.
215. WARNING: Unable to switch to thread 15
216.  
217. ==================== THREAD 16 (TCB: 0x3ffd28e0, name: 'tiT') =====================
218. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffd16a4, pvBuffer=0x3ffd2630, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
219. #1  0x4015767c in sys_arch_mbox_fetch (mbox=<optimized out>, msg=0x3ffd2630, timeout=13) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/lwip/port/esp32/freertos/sys_arch.c:325
220. #2  0x40146bc2 in tcpip_timeouts_mbox_fetch (mbox=<optimized out>, msg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/lwip/lwip/src/api/tcpip.c:110
221. #3  tcpip_thread (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/lwip/lwip/src/api/tcpip.c:148
222.  
223. ==================== THREAD 17 (TCB: 0x3ffc9bf4, name: 'cliTask') =====================
224. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffcdef0, pvBuffer=0x0, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
225. #1  0x40088800 in prvReceiveGeneric (pxRingbuffer=0x3ffcde58, pvItem1=0x3ffc98c4, pvItem2=0x0, xItemSize1=0x3ffc98c0, xItemSize2=0x0, xMaxSize=0, xTicksToWait=xTicksToWait@entry=4294967295) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_ringbuf/ringbuf.c:748
226. #2  0x40088c5c in xRingbufferReceive (xRingbuffer=0x3ffcde58, pxItemSize=0x3ffc9900, xTicksToWait=4294967295) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_ringbuf/ringbuf.c:1104
227. #3  0x4010f865 in uart_read_bytes (uart_num=0, buf=0x3ffc9940 \"\\004\", length=1, ticks_to_wait=4294967295) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/driver/uart.c:1164
228. #4  0x400f6f10 in uart_rx_char_via_driver (fd=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/vfs/vfs_uart.c:197
229. #5  0x400f686f in uart_read_char (fd=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/vfs/vfs_uart.c:239
230. #6  0x400f71a6 in uart_read (size=1, data=0x3ffae96b, fd=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/vfs/vfs_uart.c:256
231. #7  uart_read (fd=0, data=0x3ffae96b, size=1) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/vfs/vfs_uart.c:249
232. #8  0x400f78a4 in esp_vfs_read (r=<optimized out>, fd=0, dst=0x3ffae96b, size=1) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/vfs/vfs.c:446
233. #9  0x4000bdbe in ?? ()
234. #10 0x40001128 in ?? ()
235. #11 0x400594ec in ?? ()
236. #12 0x40170fde in __srget_r (ptr=0x3ffc9c60, fp=0x3ffae924) at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/rget.c:42
237. #13 0x40170a44 in _fgetc_r (ptr=0x3ffc9c60, fp=0x3ffae924) at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/fgetc.c:94
238. #14 0x40170a82 in fgetc (fp=0x3ffae924) at /builds/idf/crosstool-NG/.build/HOST-x86_64-apple-darwin12/xtensa-esp32-elf/src/newlib/newlib/libc/stdio/fgetc.c:114
239. #15 0x4010717c in linenoiseDumb (buflen=4096, prompt=<optimized out>, buf=0x3ffce1cc <error: Cannot access memory at address 0x3ffce1cc>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/console/linenoise/linenoise.c:946
240. #16 linenoise (prompt=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/console/linenoise/linenoise.c:988
241. #17 0x40104ceb in cliTask (pvParameters=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/anova-cli/anovaCLImain.c:2270
242.  
243. ==================== THREAD 18 (TCB: 0x3ffe18a4, name: 'stats') =====================
244. #0  0x4008c12a in vTaskSuspend (xTaskToSuspend=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
245. #1  0x400d55b0 in stats_task (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/oven/main/stats.c:165
246.  
247. ==================== THREAD 19 (TCB: 0x3ffbd80c, name: 'ipc0') =====================
248. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffbd388, pvBuffer=0x0, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
249. #1  0x40081e62 in ipc_task (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_ipc/ipc.c:51
250.  
251. ==================== THREAD 20 (TCB: 0x3ffbdeec, name: 'ipc1') =====================
252. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffbda68, pvBuffer=0x0, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
253. #1  0x40081e62 in ipc_task (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_ipc/ipc.c:51
254.  
255. ==================== THREAD 21 (TCB: 0x3ffd5268, name: 'wifi') =====================
256. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffd30f0, pvBuffer=0x3ffd5170, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
257. #1  0x40116351 in queue_recv_wrapper (queue=0x3ffd30f0, item=0x3ffd5170, block_time_tick=4294967295) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_wifi/esp32/esp_adapter.c:352
258. #2  0x40093314 in ppTask ()
259.  
260. ==================== THREAD 22 (TCB: 0x3ffafb5c, name: 'esp_timer') =====================
261. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffaead8, pvBuffer=0x0, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
262. #1  0x400d4d39 in timer_task (arg=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_timer/src/esp_timer.c:329
263.  
264. ==================== THREAD 23 (TCB: 0x3ffc6558, name: 'wavTaskLoop') =====================
265. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffc66d4, pvBuffer=0x3ffc64b0, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
266. #1  0x401053b3 in wavTaskLoop (pvParameter=<optimized out>) at /Users/Shared/xxxx/idf-4-2-2-apo/app/components/audio/audio.c:141
267.  
268. ==================== THREAD 24 (TCB: 0x3ffc2384, name: 'sys_evt') =====================
269. #0  0x4008dd41 in xQueueGenericReceive (xQueue=0x3ffbe174, pvBuffer=0x3ffc22a0, xTicksToWait=<optimized out>, xJustPeeking=0) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/soc/src/esp32/include/hal/cpu_ll.h:39
270. #1  0x40197814 in esp_event_loop_run (event_loop=0x3ffbe148, ticks_to_run=4294967295) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_event/esp_event.c:624
271. #2  0x40197987 in esp_event_loop_run_task (args=0x3ffbe148) at /Users/Shared/xxxx/idf-4-2-2-apo/esp-idf/components/esp_event/esp_event.c:115
272.  
273.  
274. ======================= ALL MEMORY REGIONS ========================
275. Name   Address   Size   Attrs
276. .rtc.text 0x400c0000 0x0 RW
277. .rtc.dummy 0x3ff80000 0x0 RW
278. .rtc.force_fast 0x3ff80000 0x0 RW
279. .rtc_noinit 0x50000000 0x0 RW
280. .rtc.force_slow 0x50000000 0x0 RW
281. .iram0.vectors 0x40080000 0x400 R XA
282. .iram0.text 0x40080400 0x19294 R XA
283. .dram0.data 0x3ffb0000 0x5d78 RW A
284. .noinit 0x3ffb5d78 0x0 RW
285. .flash.appdesc 0x3f400020 0x100 R  A
286. .flash.rodata 0x3f400120 0x38520 RW A
287. .flash.text 0x400d0020 0xc83c8 R XA
288. .iram0.text_end 0x40099694 0x0 RW
289. .iram0.data 0x40099694 0x0 RW
290. .iram0.bss 0x40099694 0x0 RW
291. .dram0.heap_start 0x3ffbd360 0x0 RW
292. .coredump.tasks.data 0x3ffd06b0 0x1bc RW
293. .coredump.tasks.data 0x3ffd0884 0x168 RW
294. .coredump.tasks.data 0x3ffccd14 0x168 RW
295. .coredump.tasks.data 0x3ffcc400 0x8fc RW
296. .coredump.tasks.data 0x3ffd1114 0x168 RW
297. .coredump.tasks.data 0x3ffd0f50 0x1ac RW
298. .coredump.tasks.data 0x3ffcb584 0x168 RW
299. .coredump.tasks.data 0x3ffcb240 0x32c RW
300. .coredump.tasks.data 0x3ffdfdb8 0x168 RW
301. .coredump.tasks.data 0x3ffe0b30 0x148 RW
302. .coredump.tasks.data 0x3ffdfc3c 0x168 RW
303. .coredump.tasks.data 0x3ffe0720 0x144 RW
304. .coredump.tasks.data 0x3ffbffc0 0x168 RW
305. .coredump.tasks.data 0x3ffbfe20 0x188 RW
306. .coredump.tasks.data 0x3ffc0750 0x168 RW
307. .coredump.tasks.data 0x3ffc05b0 0x188 RW
308. .coredump.tasks.data 0x3ffc72e4 0x168 RW
309. .coredump.tasks.data 0x3ffc7070 0x25c RW
310. .coredump.tasks.data 0x3ffd01f4 0x168 RW
311. .coredump.tasks.data 0x3ffd0040 0x19c RW
312. .coredump.tasks.data 0x3ffdba74 0x168 RW
313. .coredump.tasks.data 0x3ffdb8b0 0x1ac RW
314. .coredump.tasks.data 0x3ffbe9bc 0x168 RW
315. .coredump.tasks.data 0x3ffe3650 0x374 RW
316. .coredump.tasks.data 0x3ffdd13c 0x168 RW
317. .coredump.tasks.data 0x3ffdcfa0 0x184 RW
318. .coredump.tasks.data 0x3ffc160c 0x168 RW
319. .coredump.tasks.data 0x3ffc14a0 0x154 RW
320. .coredump.tasks.data 0x3ffdfa50 0x168 RW
321. .coredump.tasks.data 0x3ffdf860 0x1d8 RW
322. .coredump.tasks.data 0x3ffd28e0 0x168 RW
323. .coredump.tasks.data 0x3ffd2500 0x1c0 RW
324. .coredump.tasks.data 0x3ffc9bf4 0x168 RW
325. .coredump.tasks.data 0x3ffc9760 0x47c RW
326. .coredump.tasks.data 0x3ffe18a4 0x168 RW
327. .coredump.tasks.data 0x3ffe16f0 0x19c RW
328. .coredump.tasks.data 0x3ffbd80c 0x168 RW
329. .coredump.tasks.data 0x3ffbd670 0x184 RW
330. .coredump.tasks.data 0x3ffbdeec 0x168 RW
331. .coredump.tasks.data 0x3ffbdd50 0x184 RW
332. .coredump.tasks.data 0x3ffd5268 0x168 RW
333. .coredump.tasks.data 0x3ffd5050 0x1bc RW
334. .coredump.tasks.data 0x3ffafb5c 0x168 RW
335. .coredump.tasks.data 0x3ffaf9b0 0x194 RW
336. .coredump.tasks.data 0x3ffc6558 0x168 RW
337. .coredump.tasks.data 0x3ffc63b0 0x190 RW
338. .coredump.tasks.data 0x3ffc2384 0x168 RW
339. .coredump.tasks.data 0x3ffc2190 0x1dc RW
340.  
341. ===================== ESP32 CORE DUMP END =====================
342. ===============================================================
343. Done!

GeSHi © Codebox Plus Extension

-jw

[MicroController]

Note that the name "interrupt WDT" may be a bit confusing. As per the docs, the IWDT is "fed" from the FreeRTOS tick ISR and as such only detects if this specific ISR did not run for a certain amount of time.
Hence, the point at which the IWDT triggers may be not directly linked to the problematic code section.
In your case for example, it is unlikely that the I2C ISR runs longer than the IWDT timeout period.

Assuming your own code doesn't spend excessive time in a critical section, I think it is possible that in your application there may be periods with a lot of interrupts happening in quick succession and that this causes execution to basically jump back and forth between critical sections (queue/semaphore operations) and ISRs.

Another thing is the "spin_task"s. Name and stack trace suggest that these tasks may starve the IDLE task of CPU time, which among other things can cause FreeRTOS timer callbacks to not be processed in time.

As to the gaps in the backtrace, this does not necessarily indicate any problem. Not 100% certain about the Xtensa architecture, but on some/many platforms the call stack may fail to be correctly traced back if/when the exact stack layout is not known at build time, which can happen if/when a function allocates a dynamic amount of space from the stack at runtime (e.g. variable length arrays).

[jwwest]

Hi MC,
Thank you for your response, it has given me a couple of things to ponder in hopes to get to the bottom of this issue.

It may be quite possible that the I2C handling and the UART handling are consuming quite a bit of time and thus preventing FreeRTOS from "kicking" the IWDT. I think I will try disabling the IWDT in a test and collect some metrics from the FreeRTOS tick handler on just how long it is getting starved. Not quite sure how I to do that off the top of my head though.

As for the spin_tasks, I have enabled the FreeRTOS stats collection, but only allow it to run when for 1.5 seconds via a console command. The 2 spin_tasks sit in the suspended state otherwise.

-jw

[MicroController]

collect some metrics from the FreeRTOS tick handler

esp_register_freertos_tick_hook(...)

If you determine that the condition the IWDT detects is only temporary and normal in your application, you can of course always set a longer timeout period.