[Solved] Recipes for debugging exceptions ...

[kolban]

When we write programs and get something wrong, we see exceptions of the format:

Code: Select all

Guru Meditation Error of type IllegalInstruction occured on core   0. Exception was unhandled.
Register dump:
PC      :  400f8806  PS      :  00060130  A0      :  00000000  A1      :  3ffc9780  
A2      :  00000000  A3      :  00000000  A4      :  00000000  A5      :  00000000  
A6      :  00000000  A7      :  00000000  A8      :  800f8806  A9      :  3ffc9760  
A10     :  ffffffff  A11     :  3f408600  A12     :  3ffc97a0  A13     :  3ffbddcc  
A14     :  3ffbdde0  A15     :  00000000  SAR     :  00000011  EXCCAUSE:  00000000  
EXCVADDR:  00000000  LBEG    :  400014fd  LEND    :  4000150d  LCOUNT  :  fffffff8  
Rebooting...
ets Jun  8 2016 00:22:57

Being new to this environment, I'm looking for recipes, hints and tips on how to diagnose such problems. What debugging technologies exist? At a minimum, I'm thinking that somewhere in this story we can generate a memory map of our compiled applications and somewhere in here there is the pointer at which the failure was detected (at a guess I'll imagine that "PC" means Program Counter).

... Later ... I made some progress. Modeling ESP32 on what we'd learned for ESP8266, I tried:

Code: Select all

xtensa-esp32-elf-objdump -x app-template.elf -d

against my compiled application and, as hoped, it produced an assembly listing of the app. I could then take that listing and map that against the program counter shown in the exception ... and now I know which module within my code my exception occurred within. I think with some more practice, I might even be able to see how that relates to source statements.

[ESP_Sprite]

You have a few methods to debug stuff like this. Three of them are:

1) You didn't know this was going to happen and just have the message you mentioned.
You can use gdb to figure out what PC and (sometimes) A0 refer to. PC is indeed the Program Counter and usually is the address of the instruction causing the exception. A0, if the compiler hasn't decide to overwrite its value with something else, is the address of the instruction calling the function that crashed. Take care that because of how the Xtensa core works, the leftmost digit of the hex number may be off and you have to manually correct it to '4'. For example, for A0 = 0x800D1234, you would want to look up 0x400D1234.

Okay, say you have a PC or A0 that seems valid, what now? You can use gdb to look up what it is. (given your source tree is in the same state as when you flashed your program. If you changed stuff in the mean time, results will be off.) Say you have address 0x400f7220, you can convert it to a source code line like such:

Code: Select all

$ xtensa-esp32-elf-gdb ./build/testje.elf 
(gdb) info symbol 0x400f7220
app_main + 16 in section .flash.text
(gdb) list *0x400f7220
0x400f7220 is in app_main (/home/jeroen/esp8266/esp32/esp-idf-tests/main/./app_main.c:16).
11	    while(1);
12	}
13	
14	void app_main() 
15	{
16	    xTaskCreatePinnedToCore(unityTask, "unityTask", 4096, NULL, 5, NULL, 0);
17	}
18	

(Observe the line before the code listing: it pinpoints the problem to line 16 of app_main.c)

2) You know this is going to happen (exception is reproducible), but do not have a JTAG interface or cannot free the IO-pins for a JTAG interface.

The FreeRTOS panic handler optionally can have a small GDB stub built in. It's only good for post-mortem analysis (meaning it only kicks in after something has gone terribly wrong and you cannot really get any code running anymore after it's kicked in) but it can be extremely useful in getting a stack trace as well as inspecting the state of variables and memory to see what went wrong.

Enable it by running 'make menuconfig' and selecting 'Component config -> FreeRTOS -> Panic handler behaviour -> Invoke GDBStub'. Now run make / make flash again and reproduce the bug. On your serial terminal, you will see something like:

Code: Select all

Guru Meditation Error of type StoreProhibited occurred on core   0. Exception was unhandled.
Register dump:
PC      :  4010677e  PS      :  00060830  A0      :  800fc22c  A1      :  3ffccdb0  
A2      :  00000000  A3      :  3ffc2a74  A4      :  00000001  A5      :  3ffccdf0  
A6      :  3ffbf444  A7      :  3ffccdf0  A8      :  00000000  A9      :  00000001  
A10     :  00000001  A11     :  0000ff00  A12     :  00000000  A13     :  00060021  
A14     :  b33fffff  A15     :  00000001  SAR     :  00000004  EXCCAUSE:  0000001d  
EXCVADDR:  00000000  LBEG    :  400014fd  LEND    :  4000150d  LCOUNT  :  fffffffe  
Entering gdb stub now.
$T0b#e6

(Observe the last two lines: it indicates the panic handler is in gdbstub mode now and expects gdb to connect to it.)

Exit your serial terminal program and invoke gdb. (Change 115200 to the baud rate you use, and /dev/ttyUSB0 to the serial port the ESP32 is connected to)

Code: Select all

$ xtensa-esp32-elf-gdb ./build/testje.elf -b 115200 -ex 'target remote /dev/ttyUSB0'
Remote debugging using /dev/ttyUSB0
0x4010677e in test_func_20 () at /home/jeroen/esp8266/esp32/esp-idf-tests/components/tests/./test_panic.c:24
24		*i=1;
(gdb) bt
#0  0x4010677e in test_func_20 () at /home/jeroen/esp8266/esp32/esp-idf-tests/components/tests/./test_panic.c:24
#1  0x400fc22c in UnityDefaultTestRun (Func=0x40106774 <test_func_20>, FuncName=0x3f409b24 "Panic handler", FuncLineNum=20)
    at /home/jeroen/esp8266/esp32/esp-idf-tests/components/unity/./unity.c:1251
#2  0x400fbc96 in unity_run_single_test (test=0x3ffc2a74 <test_desc_20>) at /home/jeroen/esp8266/esp32/esp-idf-tests/components/unity/./unity_platform.c:58
#3  0x400fbcc1 in unity_run_single_test_by_index (index=0) at /home/jeroen/esp8266/esp32/esp-idf-tests/components/unity/./unity_platform.c:69
#4  0x400fbe2d in unity_run_menu () at /home/jeroen/esp8266/esp32/esp-idf-tests/components/unity/./unity_platform.c:140
#5  0x400fc2d7 in unityTask (pvParameters=0x0) at /home/jeroen/esp8266/esp32/esp-idf-tests/main/./app_main.c:10
(gdb) p i
$1 = (volatile int *) 0x0

As you can see, I can see exactly where the error happened, and I can even see that it happened because the address of the pointer 'i' was 0 when I tried to store something into it.

3) You know this is going to happen (exception is reproducible) and you have JTAG hardware and free I/O pins for the JTAG interface.

Go set up OpenOCD! Debugging stuff using a live GDB session makes life much easier; you can break at random points in code execution, change between tasks, set breakpoints/watchpoints etc. More info: https://github.com/espressif/esp-idf/bl ... penocd.rst


Anyway, hope that helps.

[kolban]

Your post was pure gold!!!

I have just practiced the first technique and it worked flawlessly ... exactly as you described. What a tremendously useful and well described technique. Over time I plan to study your post in much more depth and start learning about the other techniques. Your post will become a must read for all of us. Thank you!!!

[rudi ;-)]

Your post was pure gold!!!

1A++
txs jeroen!

[martinayotte]

+1 or even +2
Thanks, ESP_Sprite !

[rudi ;-)]

2) You know this is going to happen (exception is reproducible), but do not have a JTAG interface or cannot free the IO-pins for a JTAG interface.

The FreeRTOS panic handler optionally can have a small GDB stub built in. It's only good for post-mortem analysis (meaning it only kicks in after something has gone terribly wrong and you cannot really get any code running anymore after it's kicked in) but it can be extremely useful in getting a stack trace as well as inspecting the state of variables and memory to see what went wrong.

Enable it by running 'make menuconfig' and selecting 'Component config -> FreeRTOS -> Panic handler behaviour -> Invoke GDBStub'.

hi

where is the 'Component config -> FreeRTOS -> Panic handler behaviour -> Invoke GDBStub' option gone in menuconfig?

Panic_handler_behaviour.png (23.72 KiB) Viewed 75227 times

best wishes
rudi

[rudi ;-)]

was moved to ESP32-specific

Panic_handler_behaviour_changed_to_ESP32-specific.png (17.27 KiB) Viewed 75225 times

best wishes
rudi

[roctwo]

2) You know this is going to happen (exception is reproducible), but do not have a JTAG interface or cannot free the IO-pins for a JTAG interface.

The FreeRTOS panic handler optionally can have a small GDB stub built in. It's only good for post-mortem analysis (meaning it only kicks in after something has gone terribly wrong and you cannot really get any code running anymore after it's kicked in) but it can be extremely useful in getting a stack trace as well as inspecting the state of variables and memory to see what went wrong.

Enable it by running 'make menuconfig' and selecting 'Component config -> FreeRTOS -> Panic handler behaviour -> Invoke GDBStub'.

hi

where is the 'Component config -> FreeRTOS -> Panic handler behaviour -> Invoke GDBStub' option gone in menuconfig?

Panic_handler_behaviour.png

best wishes
rudi

Could please tell me how to insert a picture in a comment?怎么在评论里面插入图片啊？

[rudi ;-)]

Could please tell me how to insert a picture in a comment?怎么在评论里面插入图片啊？

here comes step 1 ( placed inline )

step1.png (7.71 KiB) Viewed 75056 times

here comes step 2 ( placed inline )

step 2.png (9.8 KiB) Viewed 75056 times

step1_2.png (3.63 KiB) Viewed 75056 times

hope this helps
best wishes

rudi

btw: you can insert a link to a picture to, see the format design
B i u Quote Code List List= [*] Img URL Normal Font colour
for more infos

[roctwo]

Could please tell me how to insert a picture in a comment?怎么在评论里面插入图片啊？

here comes step 1 ( placed inline )

step1.png

here comes step 2 ( placed inline )

step 2.png

step1_2.png

hope this helps
best wishes

rudi

btw: you can insert a link to a picture to, see the format design
B i u Quote Code List List= [*] Img URL Normal Font colour
for more infos

谢谢！