Disturbing loss of big packets using USB GSM modem

[dastoned]

Hi!

I am experiencing a packet loss of nearly 50% when transmitting larger packets (e.g 1400B) using a SIM7070G GSM module connected via USB OTG to ESP32S3. Smaller packets (e.g. 1000B) are OK. The terminal is lousy with errors from CDC-ACM and other USB subsystems. The exact same GSM module has a packet loss of <1% when connected to a Linux PC, so I suspect Espressif's USB drivers or modem drivers to misbehave.

HW: a SIMCOM SIM7070G evaluation board is connected to ESP32S3 DevKitC V1.1 "USB" plug using a short OTG cable. I tested the same SIM7070G EVB behind a Linux PC and it experienced no such packet loss; it has an independent power supply, good signal, correct APN, etc - so the problem does not come from GSM side.

SW: The ESP32S3 devkit runs on ESP IDF v5.1.1, with recent esp_modem and esp_modem_usb_dte components from the official component repository.

Code: Select all

dependencies:
  espressif/esp_modem: "^1.0.1"
  espressif/esp_modem_usb_dte: "^1.1.0"
  idf:
    version: "^5.1.1"

I configured the modem driver to use a single virtual USB serial port (index 2) via CMUX, gave it plenty of time to start up and connect to mobile network, etc. To verify connectivity I send around 1000 ICMP ping messages (using the ping_sock.h library in IDF) to a reliable host. An ICMP payload of 1400 bytes (i.e. the raw network packets are still far below the PPP MTU) results in packet loss of 49%:

Code: Select all

PING 1000 packets transmitted, 510 received, 49.00% packet loss, time 1246627 ms

As contrast, if I reduce the payload size to 1000 bytes, packet loss disappears:

Code: Select all

PING 1000 packets transmitted, 998 received, 0.20% packet loss, time 448171 ms

Lost ICMP responses would appear to come in from the GSM module as the USB CDC-ACM driver reports buffer overflows and the CMUX driver reports missing data:

Code: Select all

PING 1436 bytes from 185.154.221.184 icmp_seq=97 ttl=43 time=557 ms
W (11340739) cdc_acm: IN buffer overflow (22/512)
W (11340739) cdc_acm: IN buffer overflow (0/512)
W (11340749) cdc_acm: IN buffer overflow (0/512)
PING timeout from 185.154.221.184 icmp_seq=98
W (11341649) CMUX: Protocol mismatch: Missed trailing SOF, recovering...
I (11341649) CMUX: Protocol recovered
W (11341659) cdc_acm: IN buffer overflow (4/512)
W (11341669) cdc_acm: IN buffer overflow (0/512)
W (11341669) cdc_acm: IN buffer overflow (0/512)
PING timeout from 185.154.221.184 icmp_seq=99
W (11342779) cdc_acm: IN buffer overflow (32/512)
W (11342779) cdc_acm: IN buffer overflow (0/512)
W (11342789) cdc_acm: IN buffer overflow (0/512)
W (11342799) cdc_acm: IN buffer overflow (28/512)
PING 1436 bytes from 185.154.221.184 icmp_seq=100 ttl=43 time=592 ms

Note that the (22/512) are my addition to the CDC ACM driver to print the actual number of bytes transferred and entire size of the USB transfer. It kind of looks to me like the consumer of those bytes (which I assume to be PPP and/or LwIP stack) has not emptied its incoming buffer fast enough?

The packet loss wreaks havoc with my useful Internet traffic, as I see frequent TLS connection failures and slow responses when transferring larger data objects. Again, I saw no such packet loss on a Linux PC, so the prime suspect is the USB subsystem on ESP32S3.

Any advice how to diagnose or fix this?

[dastoned]

It turns out the CMUX protocol implementation is buggy over USB (I seem to recall it working fine over UART). I switched the modem from CMUX mode to plain data mode and the problems disappeared.

Code: Select all

_modem->set_mode(modem_mode::DATA_MODE)

I additionally had to raise the buffer size on the DTE to 2048 B or it refused to send out big packets.

Code: Select all

    esp_modem_dte_config_t dte_usb_config = 
        ESP_MODEM_DTE_DEFAULT_USB_CONFIG(usb_config);
    dte_usb_config.dte_buffer_size = 2048;