ESP32 crashes in monitor

[e.voellm@mail.de]

Dear developers,
with my current ESP project using IDF version v4.2-dev-1097-g2e14149bf, i have troubles to run the IDF monitor.
The issue happens reandomly on different places, also very early during startup, when my tasks are not yet running.
The core dump is mostly unusable, and this happens at totally random places, usually with IllegalInstruction or LoadProhibited reset reason.
This DOESNT happen using "putty" as serial monitor instead of the IDF monitor, and my application, well debugged runs stable that over night and also under load, but only the IDF monitor i cannot run.
Two examples here below, looks line the IDF monitor causes some memory corruption at runtime.
Played around with anything possible, nothing seems to change the things here, tried to increase memory for stack in main task, tried any debug options in menuconfig and else, but problem behaves as not related to anything than this.
Any idea how to debug this further, or fix this problem ?

Thanks in advance, Eckhard


1) Already the panic handler cores while running monitor:

Code: Select all

Rebooting...
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:4
load:0x3fff0034,len:5856
ho 0 tail 12 room 4
load:0x40078000,len:13832
ho 0 tail 12 room 4
load:0x40080400,len:4272
0x40080400: _init at ??:?

entry 0x40080658
Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.

Core  0 register dump:
PC      : 0x4049a776  PS      : 0x00060d30  A0      : 0x800d53a4  A1      : 0x3ffe3bd0  
A2      : 0x0000001a  A3      : 0x00000002  A4      : 0x00000000  A5      : 0x3ff000cc  
A6      : 0xffff8800  A7      : 0x3ff00020  A8      : 0x80125181  A9      : 0x053b0000  
A10     : 0x00000000  A11     : 0x00000000  A12     : 0x000000a0  A13     : 0x00000020  
A14     : 0x0031119a  A15     : 0xffffffff  SAR     : 0x00000007  EXCCAUSE: 0x00000000  
EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0x00000000  

Backtrace:0x4049a773:0x3ffe3bd0 |<-CORRUPTED


ELF file SHA256: 00b479f3ef733c25

Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x4009d1e3  PS      : 0x00060e33  A0      : 0x800d43c4  A1      : 0x3ffe3920  
0x4009d1e3: prvTaskGetSnapshotsFromList at /home/nextpcb/esp/esp-idf/components/freertos/tasks.c:5076
 (inlined by) uxTaskGetSnapshotAll at /home/nextpcb/esp/esp-idf/components/freertos/tasks.c:5111

A2      : 0x3ffc3018  A3      : 0x00000040  A4      : 0x3ffc34e4  A5      : 0x3ffc3510  
A6      : 0x00000000  A7      : 0x3ffbf1f8  A8      : 0x3ffc34fc  A9      : 0x3ffe3af0  
A10     : 0x3ffc3018  A11     : 0x3ffe392c  A12     : 0x00000040  A13     : 0x00000000  
A14     : 0x00000000  A15     : 0x00000000  SAR     : 0x00000007  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000000  LBEG    : 0x401a6584  LEND    : 0x401a658e  LCOUNT  : 0x00000000  
0x401a6584: uart_ll_write_txfifo at /home/nextpcb/esp/esp-idf/components/soc/src/esp32/include/hal/uart_ll.h:190
 (inlined by) uart_hal_write_txfifo at /home/nextpcb/esp/esp-idf/components/soc/src/hal/uart_hal_iram.c:40

0x401a658e: uart_hal_write_txfifo at /home/nextpcb/esp/esp-idf/components/soc/src/hal/uart_hal_iram.c:41


Backtrace:0x4009d1e0:0x3ffe3920 0x400d43c1:0x3ffe3950 0x400d4fd7:0x3ffe3980 0x400d4332:0x3ffe39d0 0x400d42d6:0x3ffe39f0 0x40097d49:0x3ffe3a40 0x40097582:0x3ffe3ab0 0x40097865:0x3ffe3af0 0x40082aae:0x3ffe3b10 0x4049a773:0x3ffe3bd0 |<-CORRUPTED
0x4009d1e0: uxTaskGetSnapshotAll at /home/nextpcb/esp/esp-idf/components/freertos/tasks.c:5111

0x400d43c1: esp_core_dump_get_tasks_snapshot at /home/nextpcb/esp/esp-idf/components/espcoredump/src/core_dump_port.c:282

0x400d4fd7: esp_core_dump_write_elf at /home/nextpcb/esp/esp-idf/components/espcoredump/src/core_dump_elf.c:586 (discriminator 2)

0x400d4332: esp_core_dump_write at /home/nextpcb/esp/esp-idf/components/espcoredump/src/core_dump_common.c:248 (discriminator 3)

0x400d42d6: esp_core_dump_to_uart at /home/nextpcb/esp/esp-idf/components/espcoredump/src/core_dump_uart.c:166

0x40097d49: esp_panic_handler at /home/nextpcb/esp/esp-idf/components/esp_system/panic.c:289

0x40097582: panic_handler at /home/nextpcb/esp/esp-idf/components/esp_system/port/panic_handler.c:498

0x40097865: xt_unhandled_exception at /home/nextpcb/esp/esp-idf/components/esp_system/port/panic_handler.c:512

0x40082aae: _xt_user_exc at /home/nextpcb/esp/esp-idf/components/freertos/xtensa/xtensa_vectors.S:697



ELF file SHA256: 00b479f3ef733c25

Re-entered core dump! Exception happened during core dump!

2) Example of a core file in a later phase

Code: Select all

Rebooting...
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x33 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:4
load:0x3fff0034,len:5856
ho 0 tail 12 room 4
load:0x40078000,len:13832
ho 0 tail 12 room 4
load:0x40080400,len:4272
0x40080400: _init at ??:?

entry 0x40080658
BTstack: setup
BTstack: execute run loop
ESP32NVS::open() handle: 0002  OK
Setup CMD begin()
ESP32NVS::getObject(SetupXC2)
Object size: 252
*blob: 0
Setup: NVS checksum ok 2916
Setup: NVS data ok
BTSender::begin() bt:1 s-bridge:1
SDP service record size: 95
Serial TX or Bridge enabled with speed: 3 baud: 19200 tx_inv: 1 rx_inv: 1
Set UART Inversion Mask (4=RX | 32=TX): 36
Serial Bluetooth bridge enabled 

eFuse Vref
Guru Meditation Error: Core  0 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x400f081c: 1f25ea1c 00f05d34 ad204136
0x400f081c: btdm_sleep_enter_phase2_wrapper at /home/nextpcb/esp/esp-idf/components/bt/controller/bt.c:860

Core  0 register dump:
PC      : 0x400f0821  PS      : 0x00060830  A0      : 0x800f115c  A1      : 0x3ffdeb40  
0x400f0821: btdm_sleep_enter_phase2_wrapper at /home/nextpcb/esp/esp-idf/components/bt/controller/bt.c:862

A2      : 0x3ffbf548  A3      : 0x000000ff  A4      : 0x3ffb8d40  A5      : 0x3ffb8bcc  
A6      : 0x004c465d  A7      : 0x3ffc23bc  A8      : 0x800f080c  A9      : 0x3ffdeb00  
A10     : 0x00000001  A11     : 0x3ffda588  A12     : 0x00000000  A13     : 0x00000000  
A14     : 0x00000000  A15     : 0x3ffda544  SAR     : 0x0000001e  EXCCAUSE: 0x00000000  
EXCVADDR: 0x00000000  LBEG    : 0x40082891  LEND    : 0x40082899  LCOUNT  : 0x00000027  
0x40082891: esp_timer_impl_get_counter_reg at /home/nextpcb/esp/esp-idf/components/esp_timer/src/esp_timer_impl_lac.c:126 (discriminator 2)

0x40082899: esp_timer_impl_get_counter_reg at /home/nextpcb/esp/esp-idf/components/esp_timer/src/esp_timer_impl_lac.c:127 (discriminator 1)


Backtrace:0x400f081e:0x3ffdeb40 0x400f1159:0x3ffdeb60 0x400f16ad:0x3ffdeb80
0x400f081e: btdm_sleep_enter_phase2_wrapper at /home/nextpcb/esp/esp-idf/components/bt/controller/bt.c:860

0x400f1159: r_rw_schedule at ??:?

0x400f16ad: btdm_controller_task at ??:?



ELF file SHA256: 0089d28105b96fb2

Core dump started (further output muted)
Received  16 kB...
Core dump finished!
Failed to run espcoredump script: Command '['/home/nextpcb/.espressif/python_env/idf4.2_py2.7_env/bin/python', u'/home/nextpcb/esp/esp-idf/tools/../components/espcoredump/espcoredump.py', u'info_corefile', u'--core', '/tmp/tmpjoI08a', u'--core-format', u'b64', '/home/nextpcb/esp/esp-idf/examples/sensor/build/sensor.elf']' returned non-zero exit status 1

[ESP_Sprite]

What power supply are you using?

[e.voellm@mail.de]

Schematic is here, its a high frequency DC/DC converter in typical application, powered either via USB or external 12V source that is also connected:
https://github.com/iltis42/OpenIVario/b ... or-NG1.pdf

[e.voellm@mail.de]

Maybe i have now an explanation, while checking power supply and find its stable with 3.31 Volts and no glitches when the problem occurs, i crosschecked all the hardware again, and found something that has to do with the handling of IO12, that i use to monitor an external switch. This IO pin i programmed as
gpio_set_pull_mode(_sw, GPIO_PULLUP_ONLY);
and once i remove this statement, things apparently work so far.
Looking at this drawing, there is a note below concerning exactly this pin that is needed to low level on startup:
https://www.espressif.com/sites/default ... eet_en.pdf
Putting together things, i suspect that ESP32 reset logic is not clearing the pull mode when resetting via monitor. I noticed that i can run well the monitor when i completely power off the board, but never when its already powered on.
My current workaround is to pull up IO12 only for milliseconds during query of my switch, then pull down again.

Please confirm this finding.

Maybe as a good improvement in IDF monitor, this issue could be taken care, means IDF monitor could reset at least IO12 to input plus pull down to ensure ESP32 got a low level on boot. Otherwise the problem happens.

[ESP_Sprite]

I think the issue here is that GPIO12 has a very high pull-up (100K) plus what I assume is a fairly large parasitic capacitance from whatever is connected to the connector GPIO12 leads up to (plus the connector and traces themselves, obviously). The monitor doesn't do anything with GPIOs, all it does is use the DTR/RTS line to pull down the EN line of the ESP32 momentarily to reset it. Normally, this time would be enough for the internal pull-up for GPIO12 to get the level there back to 3.3V, but I think the parasitic capacitance you have on that line keeps it from doing this quickly enough. The 'proper' solution would be either to lower the pulldown to e.g. 10K, or to blow the fuses in the ESP32 to always power flash from 3.3V, independent of GPIO12. (From memory, this would be XPD_SDIO_FORCE, XPD_SDIO_REG and XPD_SDIO_TIEH, please look through the esptool.py documentation to confirm that before you go blowing fuses though.)