Making an https file downloder function

[zliudr]

I'm trying to make a reliable https file downloader. I have written one based on esp_http_client_perform. The event handler has routine in the HTTP_EVENT_ON_HEADER to detect content-length and HTTP_EVENT_ON_DATA to buffer file in memory.
This works as long as the file size is less than the memory buffer. But if I want to dump the buffer to sd card so I can buffer more data, the system seems to crash. Maybe I'm not doing this correctly. Maybe while the event handler is writing to sd card, it gets reentered because more data are received?

What are the typical uses of http's event handler anyway? What do you make it do? Just dumping received data to console like that demonstrated in included examples is useless in any practical application.

The perform() seems too high of a level. I guess there's use for it such as just posting some data and not caring about details of returned header except response code and content length. So if perform is too high-level for a file downloader, then how do I write one?

I read about the other esp_http_client functions not to be used with perform(). I read the https_ota source code that does this, download in chunks and flashes OTA partition with downloads.
It calls a fetch_header() to process headers. BUT, without using perform(), how do I get the HTTP response code? fetch_header() only returns content length, which is good for file download. I could go down this route without perform if I can find the HTTP response code. Thanks.

[boarchuz]

For stream flow, see https://docs.espressif.com/projects/esp ... ttp-stream

The event handler is useful in my case to parse a very large XML file in small chunks, while keeping memory usage very low.

In your case, I could see it being useful to control the SD write. You might have a struct with the file descriptor and size_t bytes_written, which you pass via the user_data pointer in config. Use esp_http_client_read_response to receive all, which will trigger a series of events. Within each data event, write to the file and add to bytes_written. When esp_http_client_read_response is done, cleanup and return bytes_written == bytes_expected.

You might be interested in #2 here to eliminate the need for another buffer to read the response:
https://github.com/espressif/esp-idf/issues/5814

[rtborg]

@boarchuz, how do you manage to parse a segmented file on-the-fly?

I am faced with the same situation, where I need to use regex to find a value in HTML, which is way too big for fitting into RAM.