I2C crash with release/v3.0 - what's an effective way to debug this?

[WiFive]

I don't see why it would be a problem. You may want to look at the comments and code here https://github.com/espressif/arduino-esp32/issues/1093

[meowsqueak]

I don't see why it would be a problem. You may want to look at the comments and code here https://github.com/espressif/arduino-esp32/issues/1093

Thanks, I'll take some time to look through that.

It looks like I2C problems on the ESP32 are encountered fairly often and the hardware FSM bug looks like it could be to blame. I suspect in my case there's some interaction with the FSM locking at the same time as a flood of interrupts occur, but it's not clear to me what's going on.

[jgustavoam]

Hi everyone,

It seems that some obscures problems were found with I2C. Can I make an modest sugestion ?
I did my first test with I2C (using Arduino IDE) , I found that SDA and SCL Lines are very sensitive to the squaring of signals.
I do'nt know if you are using Internal pullup resistors, but in my test I used 3K3 ohms resistors to pullup both lines ( VCC=3.3V).
If I remove these resistors, I2C crash ! I sugest you use these resistors - test one device at time, then test many devices.
I hope your circuit work fine.

My tests : viewtopic.php?f=18&t=4742&p=20511&hilit ... ner#p20511

[davdav]

Hi everybody.

I'm experience the same issue on I2C (using as ESP32 master).. after some time of correct operation suddenly it crashes. I report my gdb backtrace for various cases.

#1

Code: Select all

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0)
Core 0 register dump:
PC      : 0x40089fe9  PS      : 0x00060034  A0      : 0x8008b6f3  A1      : 0x3ffb0570  
A2      : 0x3ffd3f04  A3      : 0x00000000  A4      : 0x00000002  A5      : 0x3ffc3010  
A6      : 0x00000003  A7      : 0x00060223  A8      : 0x0000cdcd  A9      : 0x0000cdcd  
A10     : 0x00000000  A11     : 0x3ffb05b8  A12     : 0x00000004  A13     : 0x3ffd3f10  
A14     : 0x00000001  A15     : 0x00000000  SAR     : 0x0000001e  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff  
Core 0 was running in ISR context:
EPC1    : 0x400d37d6  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x40089fe9

Backtrace: 0x40089fe9:0x3ffb0570 0x4008b6f0:0x3ffb0590 0x40083fb2:0x3ffb05b0 0x400843b9:0x3ffb05e0 0x400824e5:0x3ffb0610 0x400d37d3:0x00000000

Core 1 register dump:
PC      : 0x400d37d6  PS      : 0x00060334  A0      : 0x8008a5bc  A1      : 0x3ffc3610  
A2      : 0x00000008  A3      : 0x00000001  A4      : 0x00000001  A5      : 0x3ffc3144  
A6      : 0x00000000  A7      : 0x00000001  A8      : 0x3ffb3448  A9      : 0x3ffb340c  
A10     : 0x00000000  A11     : 0x00000001  A12     : 0x8008b14f  A13     : 0x3ffc3520  
A14     : 0x00000003  A15     : 0x00060023  SAR     : 0x00000000  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000  

Backtrace: 0x400d37d6:0x3ffc3610 0x4008a5b9:0x3ffc3630

Entering gdb stub now.
$T06#ba

 xtensa-esp32-elf-gdb ./build/AviorWiFiESP32.elf -b 115200 -ex 'target remote COM6'
GNU gdb (crosstool-NG crosstool-ng-1.22.0-80-g6c4433a5) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-host_pc-mingw32 --target=xtensa-esp32-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/AviorWiFiESP32.elf...done.
Remote debugging using COM6
0x40089fe9 in vPortCPUReleaseMutexIntsDisabledInternal (mux=0x3ffd3f04)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/portmux_impl.inc.h:157
157             assert(coreID == mux->owner); // This is a mutex we didn't lock, or it's corrupt
(gdb) bt
#0  0x40089fe9 in vPortCPUReleaseMutexIntsDisabledInternal (mux=0x3ffd3f04)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/portmux_impl.inc.h:157
#1  vPortCPUReleaseMutexIntsDisabled (mux=0x3ffd3f04)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/portmux_impl.h:109
#2  vTaskExitCritical (mux=0x3ffd3f04)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/tasks.c:4276
#3  0x4008b6f3 in xQueueGenericSendFromISR (xQueue=0x3ffd3ebc,
    pvItemToQueue=<optimized out>, pxHigherPriorityTaskWoken=0x3ffb05b0,
    xCopyPosition=2)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/queue.c:1281
#4  0x40083fb5 in i2c_master_cmd_begin_static (i2c_num=<optimized out>)
    at C:/msys32/home/Davide/esp/esp-idf/components/driver/i2c.c:1045
#5  0x400843bc in i2c_isr_handler_default (arg=0x3ffd3df8)
    at C:/msys32/home/Davide/esp/esp-idf/components/driver/i2c.c:375
#6  0x400824e8 in _xt_lowint1 ()
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos\xtensa_vectors.S:1105
(gdb)

#2

Code: Select all

xtensa-esp32-elf-gdb ./build/AviorWiFiESP32.elf -b 115200 -ex 'target remote COM6'
GNU gdb (crosstool-NG crosstool-ng-1.22.0-80-g6c4433a5) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-host_pc-mingw32 --target=xtensa-esp32-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/AviorWiFiESP32.elf...done.
Remote debugging using COM6
0x400843af in i2c_isr_handler_default (arg=0x3ffd3df8)
    at C:/msys32/home/Davide/esp/esp-idf/components/driver/i2c.c:373
373                     I2C[i2c_num]->int_clr.ack_err = 1;
(gdb) bt
#0  0x400843af in i2c_isr_handler_default (arg=0x3ffd3df8)
    at C:/msys32/home/Davide/esp/esp-idf/components/driver/i2c.c:373
#1  0x400824e8 in _xt_lowint1 ()
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos\xtensa_vectors.S:1105

#3

Code: Select all

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0)
Core 0 register dump:
PC      : 0x4008a165  PS      : 0x00060034  A0      : 0x8008b827  A1      : 0x3ffb0570  
A2      : 0x3ffd3f6c  A3      : 0x00000000  A4      : 0x00000002  A5      : 0x3ffd45f0  
A6      : 0x00000003  A7      : 0x00060a23  A8      : 0x00000000  A9      : 0x000000ff  
A10     : 0x00000000  A11     : 0x3ffb05b8  A12     : 0x00000004  A13     : 0x3ffd3f78  
A14     : 0x00000001  A15     : 0x00000000  SAR     : 0x00000016  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0xffffffff  
Core 0 was running in ISR context:
EPC1    : 0x400f7a50  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x4008a165

Backtrace: 0x4008a165:0x3ffb0570 0x4008b824:0x3ffb0590 0x400840e6:0x3ffb05b0 0x400844ed:0x3ffb05e0 0x40082619:0x3ffb0610 0x400f7a4d:0x00000000

Core 1 register dump:
PC      : 0x400d37de  PS      : 0x00060534  A0      : 0x8008a6f0  A1      : 0x3ffc3680  
A2      : 0x00000008  A3      : 0x00000001  A4      : 0x00000001  A5      : 0x3ffc31ac  
A6      : 0x00000000  A7      : 0x00000001  A8      : 0x3ffb3448  A9      : 0x3ffb340c  
A10     : 0x00000000  A11     : 0x00000001  A12     : 0x8008b283  A13     : 0x3ffc3590  
A14     : 0x00000003  A15     : 0x00060023  SAR     : 0x00000000  EXCCAUSE: 0x00000005  
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000  

Backtrace: 0x400d37de:0x3ffc3680 0x4008a6ed:0x3ffc36a0

Entering gdb stub now.
$T06#ba

vPortCPUReleaseMutexIntsDisabledInternal (mux=0x3ffd3f6c)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/portmux_impl.inc.h:162
162             if(mux->count == 0) {
(gdb) bt
#0  vPortCPUReleaseMutexIntsDisabledInternal (mux=0x3ffd3f6c)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/portmux_impl.inc.h:162
#1  vPortCPUReleaseMutexIntsDisabled (mux=0x3ffd3f6c)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/portmux_impl.h:109
#2  vTaskExitCritical (mux=0x3ffd3f6c)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/tasks.c:4276
#3  0x4008b827 in xQueueGenericSendFromISR (xQueue=0x3ffd3f24,
    pvItemToQueue=<optimized out>, pxHigherPriorityTaskWoken=0x3ffb05b0,
    xCopyPosition=2)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/queue.c:1281
#4  0x400840e9 in i2c_master_cmd_begin_static (i2c_num=<optimized out>)
    at C:/msys32/home/Davide/esp/esp-idf/components/driver/i2c.c:1045
#5  0x400844f0 in i2c_isr_handler_default (arg=0x3ffd3e60)
    at C:/msys32/home/Davide/esp/esp-idf/components/driver/i2c.c:375
#6  0x4008261c in _xt_lowint1 ()
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos\xtensa_vectors.S:1105

On github issue page https://github.com/espressif/esp-idf/is ... -369806882
it seems espressif guys are looking into it but no confirmation has been received.

Is it possible to know if this issue is going to be solved? Or I have to think about writing a bit-banging version of I2C to talk with a port expander without relying on i2c driver.

Thanks

[mikemoy]

Just curious about this. The OP does not give enough information to actually help you. What sensor are you trying to talk to. Are you using IDF or Arduino, What ESP32 are your using. Do you have pullup resistors on the I2C lines and what are there values. You have not shared any of your code so that we can look to see if it's something wrong with what your doing. Do you have an oscilloscope? have you looked at the I2C lines to check the signal integrity to see if it looks strange.

[meowsqueak]

Just curious about this. The OP does not give enough information to actually help you. What sensor are you trying to talk to. Are you using IDF or Arduino, What ESP32 are your using. Do you have pullup resistors on the I2C lines and what are there values. You have not shared any of your code so that we can look to see if it's something wrong with what your doing. Do you have an oscilloscope? have you looked at the I2C lines to check the signal integrity to see if it looks strange.

It's not clear who you are replying to - who is "you" in your context? Is that me? Or are you referring to OP as me, the author of the first post in this thread? Sorry, not trying to be difficult, it's just you didn't quote or address anyone so it's not clear to me who your comment is directed towards

[davdav]

@meowsqueak I think @mikemoy is referring to my last post.

To reply his questions:

- I'm talking to a TCA9535 port expander

-I'm using ESP-IDF v3.0 (77eae33a7ec6c4d42552b07f3dc2f51d0ff4e49c)

-I use 1k8 resistor as pull-up. I had 10k but checking with oscilloscope the slope was very poor. Now the signals are ok (not too much ringing on the edge). From hardware side I think everything is OK.

-my code to write and read the port expander is based on the example provided in the esp-idf folders (just adapted for TCA9535):

Code: Select all

static void i2c_example_master_init()
{
    int i2c_master_port = I2C_EXAMPLE_MASTER_NUM;
    i2c_config_t conf;
    conf.mode = I2C_MODE_MASTER;
    conf.sda_io_num = I2C_EXAMPLE_MASTER_SDA_IO;
    conf.sda_pullup_en = GPIO_PULLUP_DISABLE;
    conf.scl_io_num = I2C_EXAMPLE_MASTER_SCL_IO;
    conf.scl_pullup_en = GPIO_PULLUP_DISABLE;
    conf.master.clk_speed = I2C_EXAMPLE_MASTER_FREQ_HZ;
    i2c_param_config(i2c_master_port, &conf);

//    i2c_hw_fsm_reset(i2c_master_port);
    //ATTENTION: in function i2c_driver_install has been inserted i2c_hw_fsm_reset to reset state machine
    i2c_driver_install(i2c_master_port, conf.mode,
                       I2C_EXAMPLE_MASTER_RX_BUF_DISABLE,
                       I2C_EXAMPLE_MASTER_TX_BUF_DISABLE, 0);
}

static esp_err_t PORT_readBack(i2c_port_t i2c_num, uint8_t* data_h, uint8_t* data_l)
{
    int ret;
	//Read back sensor
    i2c_cmd_handle_t cmd = i2c_cmd_link_create();
    i2c_master_start(cmd);
    i2c_master_write_byte(cmd, TCA9535_ADDR << 1 | WRITE_BIT, ACK_CHECK_EN);
    i2c_master_write_byte(cmd, TCA9535_CMD_READ_PORTS, ACK_CHECK_EN);
    i2c_master_start(cmd);
    i2c_master_write_byte(cmd, TCA9535_ADDR << 1 | READ_BIT, ACK_CHECK_EN);
    i2c_master_read_byte(cmd, data_h, ACK_VAL);
    i2c_master_read_byte(cmd, data_l, NACK_VAL);
    i2c_master_stop(cmd);

    ret = i2c_master_cmd_begin(i2c_num, cmd, 1000 / portTICK_RATE_MS);
    i2c_cmd_link_delete(cmd);
    return ret;
}


static esp_err_t PORT_write(i2c_port_t i2c_num, uint8_t port0, uint8_t port1)
{
    int ret;
    i2c_cmd_handle_t cmd = i2c_cmd_link_create();
    i2c_master_start(cmd);
    i2c_master_write_byte(cmd, TCA9535_ADDR << 1 | WRITE_BIT, ACK_CHECK_EN);
    i2c_master_write_byte(cmd, TCA9535_CMD_WRITE_PORTS, ACK_CHECK_EN);
    i2c_master_write_byte(cmd, port0, ACK_CHECK_EN);	//high byte (output)
    i2c_master_write_byte(cmd, port1, ACK_CHECK_EN);	//low byte (output)
    i2c_master_stop(cmd);
    ret = i2c_master_cmd_begin(i2c_num, cmd, 1000 / portTICK_RATE_MS);
    i2c_cmd_link_delete(cmd);
    return ret;
}

In another task I call this function every 250ms the PORT_readBack function and every 500ms the PORT_write function.
I have also modified "i2c_driver_install" function inserting the "i2c_hw_fsm_reset" to reset the state machine.


This night I had another crash due to I2C..check below:

Code: Select all

xtensa-esp32-elf-gdb ./build/AviorWiFiESP32.elf -b 115200 -ex 'target remote COM6'
GNU gdb (crosstool-NG crosstool-ng-1.22.0-80-g6c4433a5) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-host_pc-mingw32 --target=xtensa-esp32-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/AviorWiFiESP32.elf...done.
Remote debugging using COM6
0x4000c2da in ?? ()
(gdb) bt
#0  0x4000c2da in ?? ()
#1  0x4008b319 in prvCopyDataToQueue (pxQueue=0x3ffd4f24,
    pvItemToQueue=0x3ffb05b4, xPosition=2)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/queue.c:1900
#2  0x4008b7d4 in xQueueGenericSendFromISR (xQueue=0x3ffd4f24,
    pvItemToQueue=0x3ffb05b4, pxHigherPriorityTaskWoken=0x3ffb05b0,
    xCopyPosition=2)
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos/queue.c:1193
#3  0x400840e9 in i2c_master_cmd_begin_static (i2c_num=<optimized out>)
    at C:/msys32/home/Davide/esp/esp-idf/components/driver/i2c.c:1048
#4  0x400844f0 in i2c_isr_handler_default (arg=0x3ffd4e60)
    at C:/msys32/home/Davide/esp/esp-idf/components/driver/i2c.c:378
#5  0x4008261c in _xt_lowint1 ()
    at C:/msys32/home/Davide/esp/esp-idf/components/freertos\xtensa_vectors.S:1105
(gdb)

I guess Espressif guys knows about this issue. I would like to understand if they are going to fix it soon or I have to think about
implementing by myself.

Thanks

[meowsqueak]

Looks like this might be fixed with https://github.com/espressif/esp-idf/co ... b190eff1ad

I've tested it on my boards and the problem is no longer occurring, however I'm also running a long-term test to be sure.

[Hexman64]

I'm not exactly sure which I2C problems you guys are discussing here, but I found a solution / workaround for mine on another forum. Links can be found here:
viewtopic.php?f=19&t=5094&p=22403#p22403