ESP32 UART missing RX bits fairly frequently @ 115200 230400 921600

[iosixllc]

I can see from logic analyzer (both digital and analog) the data looks correct on the line, but the ESP32 is missing bits on occasion:

The ESP32 saw: 6F 6E 6E 65 63 70 20 74 69
The correct bytes: 6F 6E 6E 65 63 74 20 74 69

uart error.PNG (115.08 KiB) Viewed 10682 times

uart error2.PNG (129.75 KiB) Viewed 10682 times

This is UART2 on pin 10 being routed via GPIO Matrix. This pin is on VSDIO/1.8V power domain and I'm on the following part:
Chip is ESP32D2WDQ5 (revision 1)
Features: WiFi, BT, Dual Core, Embedded Flash, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz

I've tested with IDF 3.3, 4.0, and 4.1beta1; this seems to be a hardware issue/glitch. I've tried putting UART ISR into RAM on 4.1beta1, but no improvement. Is it possible to route this IO pin to another module with glitch filtering and then to UART, such as MCPWM Capture or Pulse Counter? Any other suggestions of root cause?

I'll note the problem is much worse at 115200 and 230400 than 921600. I've tried going higher but at 2000000 my ESP UART0 console seems to lock up and on UART2 I get a lot of junk in receive - not sure why.

Thanks in advance for any assistance you can provide.

[ESP_Sprite]

Hm, that is a weird failure mode... you would expect that the UART misses entire bytes if there is an error in the software. How do you initialize/read the UART in software? Also, how reproducible is this error? On the hardware front, what exactly are the low and high voltage levels?

[iosixllc]

Here is the initialization:
QueueHandle_t uart_event_queue;
const uart_config_t uart_config = {
.baud_rate = 921600,
.data_bits = UART_DATA_8_BITS,
.parity = UART_PARITY_DISABLE,
.stop_bits = UART_STOP_BITS_1,
.flow_ctrl = UART_HW_FLOWCTRL_CTS_RTS,
.rx_flow_ctrl_thresh = 120,
};
uart_param_config(UART_NUM_2, &uart_config)
uart_set_pin(UART_NUM_2, 18, 10, 23, 9);
uart_driver_install(UART_NUM_2, 5120, 1024, 10, &uart_event_queue, 0);

Reading bytes:
int len = uart_read_bytes(UART_NUM_2, (uint8_t *)buffer, BUF_SIZE, 50);

This error happens at a rate of at least 1 in 20000 bits at 115200, and 1 in 100000 bits at 921600. Per the analog and digital measurements of the logic analyzer we are seeing voltage swing from 0.003V to 1.794V during the bit in question.

Another interesting tidbit - if I turn off all SPI transactions - HSPI_HOST using these pins:
busconfig.mosi_io_num = 15;
busconfig.miso_io_num = 12;
busconfig.sclk_io_num = 13;

Then no UART errors occur in a test of 8Mbit. So it does seem this is somehow related to SPI. Normally we have an SPI transaction happening in another task every 10ms.

[ESP_Sprite]

Wait, so the UART level is 1.8V? Then the issue is that GPIO34 (pin 10 of the ESP32) is in the 3.3V power domain and an 1.8V high level probably is just enough to be detected as a high... mostly, when nothing else happens that either raises the 3.3V a bit or lowers the 1.8V a bit, and in general it's out of spec as Vih at 3.3V is 2.75V. Refer to the IO_MUX appendix of the ESP32 datasheet; that table states the power domains of all I/O pins.

[iosixllc]

As mentioned, GPIO 10 is in the VSDIO power domain, which is 1.8V on this part.

[WiFive]

Are the bad bits always low? Have you tried enabling the internal pull-up?

[ESP_Sprite]

As mentioned, GPIO 10 is in the VSDIO power domain, which is 1.8V on this part.

Ah, GPIO10 is a different story. Your text stated *pin* 10, which is GPIO34.

Is this a board with a 'bare' ESP32, or are you using one of the modules (Wroom, Wrover, ...)?

[iosixllc]

No, the bad bits can be low or high - and yes internal pull-up is on - if we don't use internal pull-up we get a lot of garbage on the input. Per my plot, it doesn't look to be an electrical issue, but rather a software or hardware bug in the processor. This is a chip-down design with the part # I referenced earlier. We have over 120,000 of these in the field, with more to come.

[WiFive]

What is your level shifter and 1.8v power supply? Analog samples are taken at the esp32 pin? You could see if there are glitches when capturing samples with rmt.

[ESP_Angus]

Hi iosixllc,

Further to some of the other questions, is it possible this is a crosstalk issue with the HSPI peripheral connections?

What is the clock rate used by the HSPI peripheral?

Where was the probe connected when tracing with the Saleae? If it is a noise problem, the limited analog bandwidthof a logic analyzer is not ideal for debugging it - is it possible to get an oscilloscope capture where the probe is connected at the ESP32 end of the trace?

If you configure the HSPI peripheral to be active continuously, instead of at 10ms intervals, does the frequency of bit rates go up significantly? This may also be useful to have set when capturing a scope trace of the UART RX pin.

You can also try reducing the drive strength of the HSPI MOSI & CLK pins in case this is causing ringing and crosstalk: can call gpio_set_drive_capability(pin_num, GPIO_DRIVE_CAP_0); after the SPI bus is initialized to set the weakest drive strength on both pins, and see if this makes any difference. (Note that depending on the drive requirements on the HSPI bus, this may also cause errors on that bus - but good to get the data point in any case.)