esp-idf httpd ethernet webserver performance is slow

[PeterR]

I am using the esp-idf http webserver with HTTPD_DEFAULT_CONFIG() settings on an Ethernet wired connection.
I have a large webpage (3MB) made up of perhaps 100 files. Its a single page application.
The files are stored in a spiffs partition.
My fileserver reads file chunks of upto 10KB at a time and sends with httpd_resp_send_chunk()

I find that the webpage takes 5 seconds to load on a PC (which seems slow) but takes very very much longer on lower performance devices.
I wanted to check the factors limiting performance together with any suggestions for improving performance.

We use:
SPI Speed : 40MHz
SPI Mode : DIO

So the best performance I might expect is 10MB/s. I will of course be further (& probably significantly) limited by software latency in the SPIFF filesystem interactions with hardware e.g. finding the file, looking up location etc.
So, is SPIFF optimum for speed or should I be looking at another file system? If so what performance improvement might I achieve?

Performance degrades significantly on low performance devices. This suggests that we are choking on sockets numbers and/or packet transfer completion.
I have another Linux device serving this webpage and that device loads really quickly.
Is httpd_resp_send_chunk() blocking i.e. will I only return when the chunk has been sent and so be subject to end device performance?

I had thought that I would need to use an asynchronous server before days end, mostly to manage the data transfer latency to existing devices whilst another device loaded the webpage. I had expected performance to be much better than as is though. I have used PICs HTTP2, Cortex etc and believe I have gained much better performance. There does appear to be significant blocking/latency issues .

Any performance config setting tweaks gratefuly received.
Thanks

[DrScanlon]

Just a couple of suggestions:
1. For the smaller files--smaller than your 10K chunk buffer--I would just send them in one step with content-length in response header.
2. If you are sending chunks based on the example given in file_server.c (under protocol examples),there is a problem in that code in which the terminating chunk (0 byte count) is sent twice, leading to very sluggish content delivery. I pointed that out here:
https://esp32.com/viewtopic.php?f=2&t=11946

It's difficult to offer anything more without viewing some code.

Frank

[PeterR]

Thanks.
Good call, I will regularly send with length 0 (below) so I will fix that and post results.
In the code below response->Write() translates into httpd_resp_send_chunk()
So I read a chunk, write to a response object (httpd_resp_send_chunk()), rinse n repeat.

Code: Select all

    do {
        /* Read file in chunks into the scratch buffer */
        sizeRead = fread(chunk, 1, CHUNCK_SIZE, context->fd);

        /* Send the buffer contents as HTTP response chunk */
        if(response->Write(chunk, sizeRead) != 0) {
            fclose(context->fd);
            
            printf("File send failed!");
            /* Abort sending file */
            response->End();
            
            // @todo TEST: will this status be sent?
            response->SetStatus(HTTP_STATUS_500_INTERNAL_SERVER_ERROR);     /*-- Respond with 500 Internal Server Error --*/
            return HTTP_HANDLER_FAILED;
        }

[PeterR]

No, my bad.
I only sent size=0 once.
I had spotted the error previously and corrected but when replying had forgotten that I need to send with length=0 to end the chunking.

So, open to performance suggestions. This does seem a latency issue.

[PeterR]

I keep a running total of time spent between file open and file close along with total number of bytes transmitted.
The total amount of time spent with files open and being read is 3920mS with 856243 bytes read/sent.
(Some of that time will be spent in the http library and network stack e.g. if we run out of buffers).

The total time from GET / to last file read is 21 seconds.

I have pretty much every other task in my application closed down. When I am running my application tasks then download takes 24 seconds.

I conclude that the missing 17 seconds is within the IP stack.
I am also seeing between 200mS and 500mS between sending the last chunk of data from a file to httpd and being asked to open the next file.
There is often 200mS or so between GET requests (sequence=429) and and ACK Ack number=429
I have 54 files so the numbers seem to stack up, or at least offer the chance for significant improvement!

Any ideas?
I will create a sanitised version and post pcapngs

[DrScanlon]

I ran a quick test with a single 1.1MB file and got the following results using an 8K chunk size:
Total read/send time: 4.6 secs, measured in ESP32 code.
File read time (no sending): 2.24 secs

Transfer testing was over wifi --> router --> gigabit network --> PC. ESP32 is about 5 ft from wireless router. Tests were with WROOM32 modules with external antenna as well as printed antenna configurations. No significant difference between the two.

Results imply that the sending cost is about 2.4 secs, resulting in a net bandwidth of around 3.7Mbps. This is much lower than I expected and far below even 802.11b. I have no idea how big the link buffers are or how many there are. I'm guessing that the big delay is in emptying the active link buffer or some problem allocating buffers as you have stated. I don't think a bigger MTU would help much.

Frank

[PeterR]

Thanks, just what I was going to test next!
I am using wired Ethernet.

With a single file I transmit 233,500 bytes in 845 mS, 2.2 Mbits/S.

With 55 files I transmit 880,000 bytes in 20 seconds, I achieve approximately 330Kbits/S.

As mentioned there is upto 0.5 seconds gap between files.
There are two problems then; the gap between files and the slow transfer time.

I also have a AJAX api which uses a different http server path.
The AJAX api constructs a JSON response from application variable and sends chuncked. I run latency tests measuring interval between Javascript XML request and response. The response is a few hundred bytes long. Requests were repeated every 100 mS
I found that generally there was 50 to 80 mS latency, which is interesting.
Why am I not seeing up to 500 mS latency?

I think that you are right, there does appear to be a buffer allocation problem.
Need some ESP help!

[PeterR]

Wrote some more tests:

It takes approximately:
0.85 seconds to HTTP transfer 1MByte of const char *text using httpd_resp_send_chunk() in one 'file' go = 9.5Mbits/S*
2 seconds to read 0.9Mbytes from SPIFFS

So in theory my 0.9Mbyte webpage should load in 3 seconds but actually takes >> longer.

* Measuring to last httpd_resp_send_chunk() which probably is not the end of the transmission.
* Ethernet speed is reported as 0 (10MB) so I can improve this figure.

[gecko242]

Did you get any further with this?

I am experiencing similar issues. It seems that any requests get "stuck" in the IP stack for a long time before making it to the web server handler.

It might take 5 seconds before the server responds. What in the IP stack would cause it to hold on to a request for this length of time?

[PeterR]

I used the old school 'fs_data' FLASH based file system (there is a copy in the lwip httpd apps directory).
Create a virtual ESP32 file system. As this is a read only file system you only need fopen(), fread(), fclose() (unless your file server adds more which it shouldn't).

AFAIK the problem is with SPIFFS. Performance degraded (EDIT: exponentionally) as I used larger and larger SPIFFS partitions (see my other posts).
The latest IDF may cure. I have not bothered to check. SPIFFS/FATFS are over kill for a static HTTP file system. FATFS takes 10KB DRAM for starters.
If you write/save from your server then you can create another file system to SPIFFS/FAT/NV or whatever.

EDIT: I went from a webpage which loaded in around 100 seconds to one which loaded in <1 second over Ethernet.