Disappointing WiFi performance ESP32-S3

[robiwan]

Yes, I tried the iperf (wifi) example and I get the following data running it (log is from AP which is on a RPi5):

rpi@rpi5:~ $ iperf -s -i 3
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.4.1 port 5001 connected with 192.168.4.98 port 55390 (icwnd/mss/irtt=14/1440/4164)
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-3.0000 sec 186 KBytes 507 Kbits/sec
[ 1] 3.0000-6.0000 sec 224 KBytes 611 Kbits/sec
[ 1] 6.0000-9.0000 sec 224 KBytes 611 Kbits/sec
[ 1] 9.0000-12.0000 sec 422 KBytes 1.15 Mbits/sec
[ 1] 12.0000-15.0000 sec 229 KBytes 626 Kbits/sec
[ 1] 15.0000-18.0000 sec 332 KBytes 906 Kbits/sec
[ 1] 18.0000-21.0000 sec 297 KBytes 810 Kbits/sec
[ 1] 21.0000-24.0000 sec 226 KBytes 618 Kbits/sec
[ 1] 24.0000-27.0000 sec 390 KBytes 1.06 Mbits/sec
[ 1] 27.0000-30.0000 sec 509 KBytes 1.39 Mbits/sec
[ 1] 30.0000-33.0000 sec 464 KBytes 1.27 Mbits/sec
[ 1] 33.0000-36.0000 sec 211 KBytes 576 Kbits/sec
[ 1] 36.0000-39.0000 sec 612 KBytes 1.67 Mbits/sec
[ 1] 39.0000-42.0000 sec 485 KBytes 1.32 Mbits/sec
[ 1] 42.0000-45.0000 sec 245 KBytes 670 Kbits/sec
[ 1] 45.0000-48.0000 sec 304 KBytes 829 Kbits/sec
[ 1] 48.0000-51.0000 sec 1001 KBytes 2.73 Mbits/sec
[ 1] 51.0000-54.0000 sec 278 KBytes 758 Kbits/sec
[ 1] 54.0000-57.0000 sec 1.14 MBytes 3.18 Mbits/sec
[ 1] 57.0000-60.0000 sec 199 KBytes 543 Kbits/sec
[ 1] 60.0000-60.8690 sec 64.5 KBytes 608 Kbits/sec
[ 1] 0.0000-60.8690 sec 7.87 MBytes 1.09 Mbits/sec

I mean.. WTF?

SDK version: v5.2.3
Board: https://www.waveshare.com/wiki/ESP32-S3-Pico

[eriksl]

No, sorry, I am currently focusing on other issues. I am using WiFi mainly for short interactions, occasionally for firmware upgrades, but it's not such a big deal if that takes a few minutes.

But, yeah, it remains frustrating, as I know the performance can be better. The ESP32 really should be able to do better than the ESP8266, if only because it has two cores and a higher cpu speed (ESP32-S3). The cpu load is next to none, the access point mentions next to none errors and the connection speed is max. I have added timings to various parts of my code and none of them can explain why it's so slow. OK, I can understand it will never be the same as the 100% dedicated "iperf" tool. but 1%-10% of the speed is pathetic.

I now focus on maximising the payload size, that seems to help somewhat, even though this means the packets must be fragmented (UDP) or segmented (TCP) within LWIP.

[eriksl]

BTW do you have test results from another type of host (instead of ESP32), can it achieve higher speeds?

[robiwan]

Actually, when testing on the ESP32 again, but with UDP (iperf -u) I can get as high as 40+ Mbits/s. But I'm more interested to get a predictable and low latency between the UDP multicast packet sent, and the ESP32 receiving it. I need to verify though that the packets are sent as I think they are, I only have Wireshark on Windows to test with unfortunately...

[eriksl]

Interesting.

I have added timings to all my "handling" code and although it can account for some of the speed loss, it cannot explain the huge difference between expected speed and actually obtained speed.

[robiwan]

Running tshark on the RPi5 I can see that the packets are timely delivered:

rpi@rpi5:~ $ sudo tshark -i wlan0
Running as user "root" and group "root". This could be dangerous.
Capturing on 'wlan0'
** (tshark:3893) 16:33:27.684476 [Main MESSAGE] -- Capture started.
** (tshark:3893) 16:33:27.684549 [Main MESSAGE] -- File: "/tmp/wireshark_wlan0EFWBY2.pcapng"
1 0.000000000 192.168.4.1 → 232.10.11.12 UDP 106 35549 → 3333 Len=64
2 0.002653500 192.168.4.1 → 232.10.11.12 UDP 106 35549 → 3333 Len=64
3 0.005318816 192.168.4.1 → 232.10.11.12 UDP 106 35549 → 3333 Len=64
4 0.007983872 192.168.4.1 → 232.10.11.12 UDP 106 35549 → 3333 Len=64
5 0.010649576 192.168.4.1 → 232.10.11.12 UDP 106 35549 → 3333 Len=64
6 0.013315780 192.168.4.1 → 232.10.11.12 UDP 106 35549 → 3333 Len=64
7 0.015981854 192.168.4.1 → 232.10.11.12 UDP 106 35549 → 3333 Len=64
8 0.018647651 192.168.4.1 → 232.10.11.12 UDP 106 35549 → 3333 Len=64
9 0.021317559 192.168.4.1 → 232.10.11.12 UDP 106 35549 → 3333 Len=64
10 0.023980763 192.168.4.1 → 232.10.11.12 UDP 106 35549 → 3333 Len=64

Packet interval is very close to expected 2666 us. So the problem must lie on the ESP32 side.

[eriksl]

Well there is the whole wireless stuff still in-between... But normally the delay is minimal.

And then there is the wireless hardware of the ESP32, the wireless drivers, an IDF layer, LWIP, another IDF layer and then your application. So, who's guilty?

[DrMickeyLauer]

I can't contribute anything helpful to this discussion, but let me state that my personal issue with the WiFi performance is not the throughput as is (which is quite ok given the antenna design and the MCU price), but rather the fragility. I have been plagued by a whole lot of jitter and intermittent reconnection issues. And no, I can't just blame our design for that, since I observed the same with multiple development boards from various vendors.

[eriksl]

I think some of the havoc is due to LWIP.

Recently I learned of a well-known bug in LWIP, where TCP connections aren't "recycled" properly. I.e. if you make a lot of connections in a short time, the remote "ephemeral" port number (on the client side) will get cycled and re-used. Sometimes the old connection on the host side (ESP/LWIP) still has the connection in a "closing" state ("lingering"). The SYN packets from the client will get dropped then, causing a major delay in the connection. IIRC LWIP should simply accept the connection and implicitly finish the lingering connection.

But using UDP I have similar large delays every now and then. No traffic at all for seconds in a row while a ping that's running in the background still continues. WTF?

[robiwan]

I've tried to find WiFi related peripheral info in the ESP32-S3 technical reference manual, but there is zilch there. Anyone know where there is docs related to WiFi peripheral registers, IRQ and DMA?