HTTP client response data corrupted

ukrsms
Posts: 24
Joined: Tue Aug 20, 2019 11:30 am

HTTP client response data corrupted

Postby ukrsms » Fri Aug 23, 2019 7:57 am

I faced and issue with HTTP client when performing POST request.

The code is as followed:

Code: Select all

esp_http_client_config_t config = {
				.method=HTTP_METHOD_POST,
				.url=Request->Query,
				.timeout_ms=CLOUD_LOOKUP_TIMEOUTms,
				.disable_auto_redirect=1,					//no redirection
				.event_handler = HTTP_EventHandler,
			};
		client = esp_http_client_init(&config);
		if(!client)
		{
			break;
		}
		if(esp_http_client_set_header(client, "Content-Type", HTTP_CONT_TYPE_BIN) 	||
				esp_http_client_set_header(client, "Cache-Control", "no-cache") 	||
				esp_http_client_set_header(client, "User-Agent", HTTP_USERAGENT) 	||
				esp_http_client_set_header(client, "Accept", "*/*"))
		{
			break;
		}

		if(Err=esp_http_client_set_post_field(client, Request->BodyData, Request->BodyDataLen))
		{
			break;
		}
		Err = esp_http_client_perform(client);
		if (!Err)
		{
					Request->ReponseCode=esp_http_client_get_status_code(client);
					Request->ResponseDataLen=esp_http_client_get_content_length(client);
		}
		else
		{
			break;
		}
		if(Request->ResponseDataLen>HTTP_MAX_RESPONSE_LEN)
		{
			break;
		}
		Request->ResponseData=zalloc(Request->ResponseDataLen);
		if(!Request->ResponseData)
		{
			break;
		}
		u16 RecDataLen=0;
		if((RecDataLen=esp_http_client_read(client, Request->ResponseData, Request->ResponseDataLen/*max len*/))==-1)
		{
			break;
		}
However, after the request is finished I received response data currupted.
After studying the Esp_HTTP_client code I found that when the first data chunk is received (containing headers and body data begin) and parsed client->response->buffer->raw_data pointer points to the place where the body data begins. But the next data chunk is received before calling esp_http_client_read(). As a result new data is placed to the response buffer beginning , hence, pointer client->response->buffer->raw_data is assign to the buffer beginning causing a case that the first data chunk lays at larger address, while the second one starts at lower address (two chunks become swapped).

Could anyone tell me did I have some mistake while using the client or is there some error in it?

Update: I tried the code with both IDF versions 3.2 and 3.2.2, the result was the same.

chegewara
Posts: 2378
Joined: Wed Jun 14, 2017 9:00 pm

Re: HTTP client response data corrupted

Postby chegewara » Mon Aug 26, 2019 12:30 pm

https://github.com/espressif/esp-idf/bl ... mple.c#L55

Code: Select all

	case HTTP_EVENT_ON_DATA:
		ESP_LOGD(TAG, "HTTP_EVENT_ON_DATA, len=%d", evt->data_len);
		
		if (!esp_http_client_is_chunked_response(evt->client)) {
			// Write out data
			memcpy(response + total_len, (char*)evt->data, evt->data_len);
			total_len += evt->data_len;
			ESP_LOGD(TAG, "%d ---> %d", evt->data_len, total_len);
			// printf("%.*s", evt->data_len, (char*)evt->data);
		}

		break;

pdemianczuk
Posts: 28
Joined: Thu May 30, 2019 12:05 pm

Re: HTTP client response data corrupted

Postby pdemianczuk » Tue Dec 10, 2019 2:55 pm

Hello, I've very similar problem. I tried read 6400B (in the future I would like to read circa about 64kB)
I noticed that when I try read data less than MTU= 1440B is everything correct.
Which parameter (in Kconfig) is responsobile for buffor.
I find out the #define DEFAULT_HTTP_BUF_SIZE (512) but when i set higer I can't see anything better.

@chegewara
Unfortunetlly, when I used .event_handler I have problem with heap. When heap is above 5KB I have problem with wifi or WDT and the program is restarting.

Code: Select all


HEAP 76 kB 0x1302c                                                        
*CHECK TASK no: 1074322300       size: 2352                               
 Sent was param [0]0 [1]200 [2]1600 [3]8  progBar: 356                    
                                                                          
-esp_http_client_is_chunked_response- 0                                   
_TEST_ esp_http_client_is_chunked_response part_size 5592                 
                                                                          
-esp_http_client_is_chunked_response- 0                                   
_TEST_ esp_http_client_is_chunked_response part_size 808                  
I (25033) Check HEAP ... : free DRAM 22 kB 0x5b8c | IRAM 80 kB 0x1413c    
Sent was param [0]0 [1]200 [2]1600 [3]8  progBar: 356                     
Sent was param [0]0 [1]200 [2]1600 [3]8  progBar: 2                       
*CHECK TASK no: 1074322300       size: 2320                               
 I (25053) Check HEAP ... : free DRAM 56 kB 0xe0b0 | IRAM 113 kB 0x1c660              
                                                                          
-esp_http_client_is_chunked_response- 0                                   
_TEST_ esp_http_client_is_chunked_response part_size 5592                 

-esp_http_client_is_chunked_response- 0
_TEST_ esp_http_client_is_chunked_response part_size 808
I (25473) Check HEAP ... : free DRAM 11 kB 0x2d14 | IRAM 68 kB 0x112c4 

-esp_http_client_is_chunked_response- 0
_TEST_ esp_http_client_is_chunked_response part_size 5592

-esp_http_client_is_chunked_response- 0
_TEST_ esp_http_client_is_chunked_response part_size 808
I (25903) Check HEAP ... : free DRAM 11 kB 0x2e20 | IRAM 68 kB 0x113d0 
*CHECK TASK no: 1074322300       size: 2320
anyone knows what was happend with this componnet?

pdemianczuk
Posts: 28
Joined: Thu May 30, 2019 12:05 pm

Re: HTTP client response data corrupted

Postby pdemianczuk » Thu Dec 19, 2019 12:29 pm

Hi, I probably solve the problem. I update IDF to v4.1-dev-1533 from IDF4.0-dec-667. I had to update also toolchain (to esp-2019r2, compiler to 8.2.0), and I havn't problem with memory.
I found out some https://github.com/espressif/esp-idf/co ... 165cf29087
fix, but I am not sure is this fix the problem.
Generally, You should check is the component is updated.
I dunno know why, my app had a problem but the most people wasnt.

atlex92
Posts: 4
Joined: Fri Feb 14, 2020 2:00 pm

Re: HTTP client response data corrupted

Postby atlex92 » Fri Feb 14, 2020 2:14 pm

Hi, i had a same problem as described above, so i've managed with it by setting buffer size of client large enough, By default it will be just 512 bytes
esp_http_client_config_t config = {};
config.url = reqInfo.url.c_str();
config.is_async = false;
config.timeout_ms = reqInfo.timeoutMs;
config.buffer_size = 1024;
_client = esp_http_client_init(&config);

fkremer
Posts: 2
Joined: Thu Jun 25, 2020 10:52 am

Re: HTTP client response data corrupted

Postby fkremer » Thu Jun 25, 2020 10:55 am

atlex92 wrote:
Fri Feb 14, 2020 2:14 pm
Hi, i had a same problem as described above, so i've managed with it by setting buffer size of client large enough, By default it will be just 512 bytes
esp_http_client_config_t config = {};
config.url = reqInfo.url.c_str();
config.is_async = false;
config.timeout_ms = reqInfo.timeoutMs;
config.buffer_size = 1024;
_client = esp_http_client_init(&config);
Thank you very much that solved my problem :)

karunt
Posts: 94
Joined: Sat Apr 03, 2021 7:58 am

Re: HTTP client response data corrupted

Postby karunt » Fri May 07, 2021 3:10 pm

I'm having a similar issue. I'm trying to get data from an image file in hexdecimal format. When the node.js server converts the image file to hexdecimal format, the data size increases drastically (9034 bytes, for example). What's the max buffer size I can set in esp_http_client.h (DEFAULT_HTTP_BUF_SIZE) so that esp_http_client will receive the entire chunk in one go as opposed to several small chunks? It appears that esp_http_client_read() eventually triggers the lwip TCP/IP function "lwip_rec_tcp" in lwip/src/api/sockets.c, which retrieves data with a do ... while loop, but all the small chunks of data are stored in a small sized buffer. The problem I have with receiving data in small chunks is that I'm unable to write the small chunks to a spiffs file to use later. Only the last chunk eventually gets returned to esp_http_client_read() and is recordable in the spiffs file (which I invoke after esp_http_client_read is finished). Any help in this matter would be appreciated. Do I have to save each chunk of data within the lwip_rec_tcp function's do ... while loop itself? I was hoping not to touch the lwip component files (specifically lwip/src/api/sockets.c) at all.

karunt
Posts: 94
Joined: Sat Apr 03, 2021 7:58 am

Re: HTTP client response data corrupted

Postby karunt » Fri May 14, 2021 7:53 am

For those stuck with this issue, I realized the correct way to go about this is to use ESP's http_native_request function in the http protocol example. There, you can put the esp_http_client_read_response function in a while loop and output_buffer will capture all the server data with each iteration of the while loop.

Here's my setup:

Code: Select all

static void http_native_request(void)
{
    char output_buffer[MAX_HTTP_OUTPUT_BUFFER] = {0};   // Buffer to store response of http request
    int content_length = 0;
    int track_length = 0;
    int max_buff = 2048;
    esp_http_client_config_t config = {
        .url = "http://192.168.1.199/api?file=abc",
    };
    esp_http_client_handle_t client = esp_http_client_init(&config);

    // GET Request
    esp_http_client_set_method(client, HTTP_METHOD_GET);
    esp_err_t err = esp_http_client_open(client, 0);
    if (err != ESP_OK) {
        ESP_LOGE(TAG, "Failed to open HTTP connection: %s", esp_err_to_name(err));
    } else {
        content_length = esp_http_client_fetch_headers(client);
        track_length = content_length;
        if (content_length < 0) {
            ESP_LOGE(TAG, "HTTP client fetch headers failed");
        } else {
            //adding new code
            do {
            int data_read = esp_http_client_read_response(client, output_buffer, max_buff);
            if (data_read >= 0) {
                ESP_LOGI(TAG, "HTTP GET Status = %d, content_length = %d",
                esp_http_client_get_status_code(client),
                esp_http_client_get_content_length(client));
                //ESP_LOG_BUFFER_CHAR(TAG, output_buffer, strlen(output_buffer));
                track_length -= data_read;
                if (max_buff > track_length){
                    max_buff = track_length;
                }
                create_file_app(output_buffer);
                ESP_LOGI(TAG, "max_buff = %d, track_length = %d \n", max_buff, track_length);
            } else {
                ESP_LOGE(TAG, "Failed to read response");
            }
            } while (
                track_length>0
            );
        }
    }
    esp_http_client_close(client);
}

Who is online

Users browsing this forum: No registered users and 119 guests