ESP32 Forum

Posted: **Thu Jun 20, 2024 9:58 am**

Hi,
I have a rather large programm for the ESP8266 that needs beside WiFi two additional credentials that I currently adjust before compiling.
Compiling (and adjusting all libraries, matching the device, etc...) is a tough task for many beginners.
I would love to be able to distribute a common runtime loaded with ESPtool.py without asking the user to compile.

The WiFi data are given using Smartconfig, because it does not need additional libraries.
(I am close to the max Program memory)

Entering the two additional credentials over a Web-interface just requires too much additional resources, since you need all the HTML code as string in program space.
Does anyone know how to pass these two parameters
-using ESPTools.py,
-or similar script to write parameters in EEProm before uploading
-or using Serial after uploading?

Thank you for any clue…

Posted: **Thu Jun 20, 2024 1:48 pm**

rin67630 wrote: ↑
Thu Jun 20, 2024 9:58 am
Hi,
I have a rather large programm for the Raspberry Pi that needs beside WiFi two additional credentials ...
Thank you for any clue…

I'm assuming that you meant "ESP8266" in your first sentence... or you wouldn't be here?

I've been experimenting with dynamically changing the binary code on the ESP after installation. I find it works fine. I have tried changing the binary on the development machine before uploading. You could have some install program prompt the end-user for the credentials, make the substitution in the binary and then start the upload process. Just a thought.

Posted: **Thu Jun 20, 2024 5:51 pm**

Inq720 wrote: ↑
Thu Jun 20, 2024 1:48 pm
I'm assuming that you meant "ESP8266" in your first sentence...

True, now corrected.
I dont need to change the code, just to enter two credentials strings in EEPROM and differenciate between the uninitialized state and the rebbot with existing EEPROM data.
Before uploading or after uploading, worst case over a serial dialog.
OK, i could write and debug the code for the last option, but if I could avoid reinventing the wheel?
[/quote]

Posted: **Thu Jun 20, 2024 11:41 pm**

rin67630 wrote: ↑
Thu Jun 20, 2024 5:51 pm
True, now corrected.

I figured as much, I looked into your Karajan project and figured where you're going with this question...

rin67630 wrote: ↑
Thu Jun 20, 2024 5:51 pm
I dont need to change the code, just to enter two credentials strings in EEPROM and differenciate between the uninitialized state and the rebbot with existing EEPROM data.
Before uploading or after uploading, worst case over a serial dialog.
OK, i could write and debug the code for the last option, but if I could avoid reinventing the wheel?

It is possible that I didn't get the full gist of your question OR you didn't get the full gist of my response. I assume you want to have some user friendly UI to allow some novice user, on their development machine, to enter the credentials of their router and possibly other data proprietary to your library. I tried to address that concern. If I was wrong, please elaborate and I'll do the same.

What I was suggesting would bypass SmartConfig entirely and thus would be "re-inventing the wheel." But it doesn't require any web server type library (like my library). It is as simple as the wheel and requires very little program/data space:

(1) Declare and instantiate a struct of the pertinent data.

Code: Select all

struct InqConfig
{
  char hostSSID[32];  
  char hostPW[32];  
  u16 port;
  // ... other data you wish to define
  char apiPW[32];
};

InqConfig test = {"Gonzo", "Was here!", 42, "You seek the Holy Grail"};

void setup()
{
  Serial.begin(74880);
  delay(1000);
  Serial.println("");
  Serial.printf("%s, %s, %u, %s\n", 
    test.hostSSID, test.hostPW, test.port, test.apiPW);
}

void loop()
{
}

(2) Compile it for your distribution.
(3) Create an install script using "PowerShell" on Windows or "sed" on Linux to replace text in the binary with the actual users proper credentials. Here is the binary data displayed of the bin file showing the struct within the binary.

: before.png (35.25 KiB) Viewed 5612 times

(4) Using the PowerShell/sed make the changes in the binary file data. I used the binary editor above for the test.
(5) Upload the modified bin using ESPtool.py as you planned to do.
(6) The Sketch uses the value now defined in the modified struct.

I assure you it works. It is also more secure because no hacker is looking for the credentials in the program space. I actually encrypt them in my program space so they can't even be found if the hacker knows the password. It simply looks like binary code.

Posted: **Fri Jun 21, 2024 3:11 am**

Inq720 wrote: ↑
Thu Jun 20, 2024 11:41 pm
It is as simple as the wheel and requires very little program/data space:
(1) Declare and instantiate a struct of the pertinent data. ...
(2) Compile it for your distribution.
(3) Create an install script using "PowerShell" on Windows or "sed" on Linux to replace text in the binary with the actual users proper credentials.

Wow! I could not figure it were that simple, just fiddling into the binary to replace a string
Doesn't the .bin have a checksum?
Have you got an example of your script? I suppose that it must strictly overwrite the Strings, keeping the length.
A plain text substitution, as a text editor does, could potentially change the string size?

In fact i'm not asking for Karajan, which is a framework addressed to programmers anyway, but for e.g. https://github.com/rin67630/Sound-press ... er-Booster which bases on Karajan and is addressing end-users.

Thank you for your help and regards
Laszlo.

Posted: **Fri Jun 21, 2024 11:52 am**

rin67630 wrote: ↑
Fri Jun 21, 2024 3:11 am
Wow! I could not figure it were that simple, just fiddling into the binary to replace a string
Doesn't the .bin have a checksum?
Have you got an example of your script? I suppose that it must strictly overwrite the Strings, keeping the length.
A plain text substitution, as a text editor does, could potentially change the string size?

In fact i'm not asking for Karajan, which is a framework addressed to programmers anyway, but for e.g. https://github.com/rin67630/Sound-press ... er-Booster which bases on Karajan and is addressing end-users.

Thank you for your help and regards
Laszlo.

No checksum... at least for now. As you probably know... it changes all the time. They currently appear to compress the binary during upload and do a CRC then to make sure it gets uploaded correctly. The ESP isn't going to do a binary CRC just to see if a binary is valid at runtime. Can't waste the time/memory. It just pukes if its bad.

Yes, you are correct. File length can't be changed. It must do a binary search and replace with exact length, location and null termination. If you are modifying integers, you have to make sure to maintain proper Endian and byte packing.

I don't actually do it with scripts. I do it with code on the ESP, but I don't have this out in any published code/binary. From purely a curiosity/academic perspective, I was wanting to see if I could use some old-timer technique (80286 protected, virtual address space mode) to use the full 4MB of the ESP as program space. Not that I need more than 1MB, but just as an intellectual challenge. Even my 40K lines library fits on 512K ESPs with room for user coding. This first step doing simple data replacement proved to me that it is feasible.

Secondarily, I was considering using it for keeping WiFi credentials. As the ESP8266 is pretty weak on security, hiding that data in program space at random locations is far harder to find and dig out than the standard methods that hide it on the EEPROM sector devoted for that purpose. That is why I do the search and replace on ESP... so that the end-user can add/del/edit WiFi connections at will.

Hope the idea supports your needs (or spawns other creative uses).

Good luck,
Inq

Posted: **Sun Jun 23, 2024 12:56 pm**

Inq720 wrote: ↑
Fri Jun 21, 2024 11:52 am
Yes, you are correct. File length can't be changed. It must do a binary search and replace with exact length, location and null termination. If you are modifying integers, you have to make sure to maintain proper Endian and byte packing.

That appears to be the tricky part of the game.
Of course one can do it with a Hex editor, but since that has to be done by noobs, a script would have been the right way to do.
In fact the replacement should consider four strings only:
1) SSID,
2) WLAN Password,
3) User name in the cloud,
4) Credentials of the cloud.
The remaining configuration parameters can be retrieved from the cloud service upon setup, solving the endian encoding problem.

Posted: **Sat Jun 29, 2024 9:56 am**

I thought I might utilize your OP premise for end-user installation and if you haven't found a better method, here is quick and dirty utility you can include with your install script to do the string replacements in the bin file before uploading. Developed on a RasPi assuming from your first post that is your targeted end-user installing machine. Should compile on Windows.

Code: Select all

// BinaryChange.cpp 
// Copywrite:  2024 Inq (InqOnThat.com) No rights reserved.  No warranty provided.
 
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>

char* find(char* haystack, int sizeHaystack, char* needle)
{
    char* rtn = haystack;
    char* last = haystack + sizeHaystack;
    do
    {
        rtn = (char*)memchr(rtn, *needle, last - rtn);
        if (!rtn)
            return NULL;
        char* fnd = strstr(rtn, needle);
        if (fnd)
            return fnd;
        rtn++;
    } while (rtn < last);
    
    return NULL;
}

int main(int argc, char *argv[]) 
{
    if (argc != 4)
    {
        printf("usage:  ./BinaryChange fileName searchFor changeTo\n");
        return 1;
    }
    printf("File '%s' Replace '%s' with '%s'.\n", argv[1], argv[2], argv[3]);
    
    int fd = open(argv[1], O_RDONLY);
    if (!fd)
    {
        printf("File not found\n");
        return 2;
    }
    int length = lseek(fd, 0, SEEK_END);
    printf("File size = %d\n", length);
    
    char* data = (char*)malloc(length);
    lseek(fd, 0, SEEK_SET);
    read(fd, data, length);
    close(fd);
    
    char* pt = find(data, length, argv[2]);
    if (pt)
    {
        strcpy(pt, argv[3]);
        printf("One substitution made at %d.\n", pt - data);

        fd = open(argv[1], O_CREAT | O_WRONLY);
        write(fd, data, length);
        close(fd);
    }
    else
    {
        printf("Token not found.\n");
        return 3;
    }
    return 0;
}

Posted: **Sat Jun 29, 2024 5:22 pm**

Inq720 wrote: ↑
Sat Jun 29, 2024 9:56 am
...here is quick and dirty utility you can include with your install script to do the string replacements in the bin file before uploading...

A warm thank you for the example.

In fact, since the binary has to be loaded with the python script esptool.py, I figured to modify esptool.py into esppatch.py to make the substitutions just before uploading.
So the end user has nothing to compile.

Just use Python (which he needs anyway) start esppatch.py with the executable as argument, enter the credentials and upload.

Just that I have no real experience in Python

Posted: **Sat Jun 29, 2024 5:48 pm**

rin67630 wrote: ↑
Sat Jun 29, 2024 5:22 pm

Inq720 wrote: ↑
Sat Jun 29, 2024 9:56 am
...here is quick and dirty utility you can include with your install script to do the string replacements in the bin file before uploading...
A warm thank you for the example.
I will need some time to undestand it. I hope I can elaborate on it.

I've never created an install package for Linux, but if it's like Windows, the installer GUI can prompt the user for the things you want like SSID and passphrase and the GUI would just run the BinaryChange executable (after you compile and put in into the install package) to be used during setup.

In the static struct illustrated above, just make sure you use unique replacement strings so they won't be somewhere else in the bin file. ie...

Code: Select all

struct PersistData
{
  char hostSSID[32];  
  char hostPW[32];  
};

PersistData data = {"UserSSID", "UserPassword"};

After you prompt for end-user's data, you'd have the script run:

./BinaryChange YourAppBinary.bin UserSSID ThereHomeRouterSSID
./BinaryChange YourAppBinary.bin UserPassword ThereGobballyGook

Have script upload to their ESP and it should connect strait away.

BTW - If you know anything about making install programs for Linux, I'd appreciate any suggestions, recommended tutorials, etc.

ESP32 Forum

Passing two credentials at initial boot over serial, not a Website

Passing two credentials at initial boot over serial, not a Website

Re: Passing two credentials at initial boot over serial, not a Website

Re: Passing two credentials at initial boot over serial, not a Website

Re: Passing two credentials at initial boot over serial, not a Website

Re: Passing two credentials at initial boot over serial, not a Website

Re: Passing two credentials at initial boot over serial, not a Website

Re: Passing two credentials at initial boot over serial, not a Website

Re: Passing two credentials at initial boot over serial, not a Website

Re: Passing two credentials at initial boot over serial, not a Website

Re: Passing two credentials at initial boot over serial, not a Website