Downloading binary files with AT+HTTPCGET

[tjaffey]

With ESP-AT, is it possible to download binary files using "AT+HTTPCGET"?

It seems that the "+AT+HTTPCGET" responses are ASCII with non-printing characters stripped out.
Is it possible to request ESP-AT to pass through the raw binary data escaped in some way?

I understand that I could use the passthrough mode to open a raw SSL socket to a webserver, but then my host processor would need to implement an HTTP client.

As a workaround, I am currently Base64 encoding the binary on the server and decoding on my host processor.
However, this adds overhead and means it's not possible to request arbitrary binary files from HTTP servers.

[esp-at]

i think AT+HTTPCGET supports to download binary files. Could you please provide the reproduce steps, thanks.

~

[tjaffey]

I cannot receive binary data. The ESP-AT code appears to drop non-printing characters.

I produced a binary file using this short C program. It contains all of the characters from 0-255, repeated for 4 cycles:

Code: Select all

#include <stdio.h>
#include <stdint.h>
void main(void) {
    uint8_t c = 0;
    int i;

    for (i=0;i<1024;i++) {
        printf("%c", c++);
    }
}

I then request the file using:

Code: Select all

AT+HTTPCGET="http://chuck.local:8080/raw/foo.bin",2048,2048

Here is the data I get from ESP-AT, both in ASCII (RXLINE) and in hex (RXLINERAW).

Code: Select all

RXLINE: 'AT+HTTPCGET="http://chuck.local:8080/raw/foo.bin",2048,2048'
RXLINERAW:41542b48545450434745543d22687474703a2f2f636875636b2e6c6f63616c3a383038302f7261772f666f6f2e62696e222c323034382c32303438
RXLINE: '+HTTPCGET:1024,'
RXLINERAW:2b48545450434745543a313032342c

The first bytes received are 0x20, 0x21 - printable ASCII characters. The non-printing binary characters seem to be discarded.

Code: Select all

RXLINE: ' !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~'
RXLINERAW:202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e
RXLINE: ' !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~'
RXLINERAW:202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e
RXLINE: ' !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~'
RXLINERAW:202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e
...

[tjaffey]

Using these instructions, it's possible to place the ESP-AT device into a passthrough mode where there are no encoding issues.

https://github.com/espressif/esp-at/blo ... s.rst#L159

However, it's not clear how to detect the end of a file transfer. If the remote web server does not support HEAD requests (to fetch the expected Content-Length), how can I detect that the file transfer has completed?

[tjaffey]

Using the above instructions it's possible to have ESP-AT pass through most data, but it seems not all.

I have a webserver with a file containing the following 9 bytes 0d0a0d0a0d0a0a0d0a ("\r\n\r\n\r\n\n\r\n"). ESP-AT appears to be doing some filtering which ends the transfer when "\n\n" is encountered.

Code: Select all

AT+HTTPCGET=http://my.server.com/foo.bin,2048,2048

Returns only 4 bytes before ending the transfer

Code: Select all

0d0a
0d0a
0d0a4f4b0d0a ("\r\nOK\r\n")

How can this be fixed? ESP-AT appears unable to transfer arbitrary binary files.

[esp-at]

could you please enable AT logs and compile AT firmware to see what happened:

python build.py mencuonfig -> Component config -> Log output -> Default log verbosity -> Info
python build.py mencuonfig -> Component config -> AT -> Enable ESP-AT Debug
python build.py mencuonfig -> Component config -> AT -> Enable ESP-AT Debug -> Enable Network Debug

https://docs.espressif.com/projects/esp ... le_it.html

If you have difficulties in compiling ESP-AT project, could you please provide your AT+GMR response and foo.bin here. i would like to look into it.

[tjaffey]

'AT+GMR'
'AT version:3.5.0.0-dev(0f91d98 - ESP32C3 - Jul 19 2024 02:57:45)'
'SDK version:v5.0.6-dirty'
'compile time(5d1f30af):Aug 6 2024 09:58:55'
'Bin version:v3.4.0.0-dev(MINI-1)'

Log file attached. The zip file contains the binary data I am attempting to download.

Thanks

[esp-at]

thanks for your example, i can reproduce it now. will fix it within one week.

~

[esp-at]

Hello, this issue should get fixed on master branch, could you please git pull the code and have a test.

[tjaffey]

Thank you. This is working in all of my tests so far.