simple standalone bootloader suitable for customers / end users?

OBDave · Postby **OBDave** » Wed May 08, 2019 10:50 pm

Hello,

Those of you who have shipped ESP32-based devices, how do you distribute new firmware to end users?

I am currently evaluating ESP32 as a potential solution for new product development, but I need to figure out if there's a standalone flash utility that end users can use to flash in firmware updates.

Some background: my current products are MIDI (musical instrument) devices based on PIC18xx microcontrollers. I ended up writing a MIDI bootloader and flash utilities for Windows and Mac so that end users could flash in the latest firmware using a generic USB-MIDI dongle, with code updates being posted on the company website. For a future product, I am considering moving to an ESP32 platform so that users can configure their devices using a web page served up by an esp32 ppt server over a soft AP. I just started playing with the esp32-wifi-manager to see if I can modify that to suit this purpose.

But a hard requirement is that customers need to be able to download new firmware from our website and flash it in to pick up new features, bug fixes, etc. And asking them to pip install a bunch of command line tools is asking way too much for most end users. It needs to be a dead-simple GUI app available for both Mac and Windows, but I haven't been able to find any. Do these tools exist? How do people manage this problem currently? Is nobody shipping user-upgradeable esp32-based products yet?

Many thanks,

Dave

WiFive · Postby **WiFive** » Thu May 09, 2019 12:40 am

https://github.com/search?q=esp32+ota
https://github.com/doayee/esptool-esp32-gui

OBDave · Postby **OBDave** » Thu May 09, 2019 12:46 am

Awesome, thank you!

OBDave · Postby **OBDave** » Sun May 26, 2019 7:44 pm

Ok, well I have made a ton of progress on the web server aspect of my project and now that I'm revisiting this requirement, I think I need to make my request a little more specific.

My project operates as an Access Point serving up web pages. I'm using the ESP http server, and I have it serving up html and css pages. That all works great. It will never be connected to a network so I am not using the http client. The examples cited above have the ESP32 device run a web client to go fetch the firmware from elsewhere on the web. That won't work for my application.

In my use case, the end user would have previously downloaded updated firmware from our website then use a computer to connect to the ESP32 device via its Wifi hotpot, navigate to a web page that it's serving and then upload a file. But it appears that the http server has a buffer limitation that prevents it from receiving large HTTP chunks. And I haven't even gotten to the point where I need to figure out the bootloader aspect of it. So that's one roadblock. I may not be entering the right search terms, but I have not been able to find any examples remotely close to what I want to do.

I also looked at the GUI bootloader, and a least on the Mac version, the app is unsigned, so I'd be asking non-technical end users to tinker with their security settings, and in my case, ultimately to find that the app crashes at init under El Capitan. So I'm stuck there as well.

Any suggestions would be greatly appreciated. Thanks in advance.

Dave

Postby **ESP_Sprite** » Mon May 27, 2019 1:59 am

You should be able to connect OTA to the webserver... can we see your code there so we can see what's going wrong there?

OBDave · Postby **OBDave** » Mon May 27, 2019 3:50 am

Okay, that's probably a good idea.

I made a stripped-down version of my application. The only thing in components.mk is the index.html file which is here.

Code: Select all

<!DOCTYPE html>
<head>
	<title>Sample web server</title>
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
</head>

<body>
	<h2>Dropdown Menu (this works)</h2>
	<form id="dropdown_menu" method="post" action="">
		<select class="element select medium" name="item-choice"> 
			<option value="item1" >item 1</option>
			<option value="item2" >item 2</option>
			<option value="item3" >item 3</option>
			<option value="item4" >item 4</option>
			<option value="item5" >item 5</option>
		</select>
		<label for="item-choice">select an item</label> <BR><BR>
		<input type="hidden" name="form_id" value="choices" />
		<input id="saveForm" class="button_text" type="submit" name="submit" value="submit" />
	</form>	
	<br/>
	<hr/>

	<h2>File Upload (this blows up)</h2>
	<p>Select a file to upload and look what happens in the log</p>
	<BR>

	<form method="post" enctype="multipart/form-data">
		<input type="file" name="name">
		<input class="button" type="submit" value="Upload">
	</form>
</body>
</html>

And the entire application in main.c is as follows. As you can see I started with the persistent sockets example, which seemed like a decent enough starting point. It uses the http-server library, which not all of the examples do. http-server provides some nice tools.

Code: Select all

/* Persistent Sockets Example

   This example code is in the Public Domain (or CC0 licensed, at your option.)

   Unless required by applicable law or agreed to in writing, this
   software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
   CONDITIONS OF ANY KIND, either express or implied.
*/

#include <esp_wifi.h>
#include <esp_event_loop.h>
#include <esp_log.h>
#include <esp_system.h>
#include <nvs_flash.h>
#include <string.h>
#include <esp_http_server.h>


#define	CONFIGURE_AS_SOFT_AP	1

#if CONFIGURE_AS_SOFT_AP
	#define EXAMPLE_WIFI_SSID "ESP32_WIFI"
	#define EXAMPLE_WIFI_PASS "12345678"
#else
	#define EXAMPLE_WIFI_SSID CONFIG_WIFI_SSID
	#define EXAMPLE_WIFI_PASS CONFIG_WIFI_PASSWORD
#endif

static const char *TAG="APP";

/* embedded binary data */
extern const char index_html_start[] asm("_binary_index_html_start");
extern const char index_html_end[] asm("_binary_index_html_end");

const static char http_204[] = "HTTP/1.1 204 No Content\nContent-Length: 0\n\n";


esp_err_t index_get_handler(httpd_req_t *req)
{
    ESP_LOGI(TAG, "retrieving index.html");

    httpd_resp_send(req, index_html_start, index_html_end - index_html_start);
    return ESP_OK;
}

esp_err_t index_post_handler(httpd_req_t *req)
{
	char buf[256];
	int  ret;

	char temp[64];

	ESP_LOGI(TAG, "post handler for index.html");

	(void) memset(buf, 0, sizeof(buf));

	/* Read data received in the request */
	ret = httpd_req_recv(req, buf, sizeof(buf));
	if (ret <= 0) 
	{
		if (ret == HTTPD_SOCK_ERR_TIMEOUT) 
		{
			httpd_resp_send_408(req);
		}
		return ESP_FAIL;
	}

	ESP_LOGI(TAG, "I think what was posted was [%s]", buf);

	(void) memset(temp, 0, sizeof(temp));

	ret = httpd_query_key_value(buf, "submit", temp, sizeof(temp));
	if (ret == ESP_OK)
	{
		/* subnmit method was used in the http post   */
		ESP_LOGI(TAG, "detected submit [%s]", temp);

		ret = httpd_query_key_value(buf, "item-choice", temp, sizeof(temp));
		if (ret == ESP_OK)
		{
			ESP_LOGI(TAG, "you have chosen [%s]", temp); /* This all works great... */
		}
	}

	httpd_resp_set_status(req, HTTPD_204);
	httpd_resp_send(req, http_204, sizeof(http_204));
	return ESP_OK;
}

static unsigned visitors = 0;  /* I don't know why the httpd_uti struct requires this.... */

httpd_uri_t index_get = 	{ .uri      = "/", 		.method   = HTTP_GET, 	.handler  = index_get_handler, 		.user_ctx = &visitors };
httpd_uri_t index_get2 = 	{ .uri      = "/index.html", 	.method   = HTTP_GET, 	.handler  = index_get_handler, 		.user_ctx = &visitors };
httpd_uri_t index_post = 	{ .uri      = "/", 		.method   = HTTP_POST, 	.handler  = index_post_handler, 	.user_ctx = &visitors };


httpd_handle_t start_webserver(void)
{
    httpd_config_t config = HTTPD_DEFAULT_CONFIG();
    // Start the httpd server
    ESP_LOGI(TAG, "Starting server on port: '%d'", config.server_port);
    httpd_handle_t server;

    if (httpd_start(&server, &config) == ESP_OK) {
        // Set URI handlers
        ESP_LOGI(TAG, "Registering URI handlers");

        httpd_register_uri_handler(server, &index_get);
        httpd_register_uri_handler(server, &index_get2);
        httpd_register_uri_handler(server, &index_post);
        return server;
    }
    ESP_LOGI(TAG, "Error starting server!");
    return NULL;
}


void stop_webserver(httpd_handle_t server)
{
    // Stop the httpd server
    httpd_stop(server);
}



static esp_err_t event_handler(void *ctx, system_event_t *event)
{
    // httpd_handle_t *server = (httpd_handle_t *) ctx;

    ESP_LOGI(TAG, "event_handler received id %d", event->event_id);

    switch(event->event_id) {
    case SYSTEM_EVENT_STA_START:
        ESP_LOGI(TAG, "SYSTEM_EVENT_STA_START");
        ESP_ERROR_CHECK(esp_wifi_connect());
        break;
    case SYSTEM_EVENT_STA_GOT_IP:
    case SYSTEM_EVENT_AP_STAIPASSIGNED:
        ESP_LOGI(TAG, "SYSTEM_EVENT_STA_GOT_IP");
        ESP_LOGI(TAG, "Got IP: '%s'",
                 ip4addr_ntoa(&event->event_info.got_ip.ip_info.ip));
        break;
    case SYSTEM_EVENT_STA_DISCONNECTED:
        ESP_LOGI(TAG, "SYSTEM_EVENT_STA_DISCONNECTED");
        ESP_ERROR_CHECK(esp_wifi_connect());
        break;
    default:
        break;
    }
    return ESP_OK;
}

#if CONFIGURE_AS_SOFT_AP
	static tcpip_adapter_ip_info_t info;
#endif

static void init_wifi_and_httpd(void *arg)
{
	tcpip_adapter_init();
	ESP_ERROR_CHECK(esp_event_loop_init(event_handler, arg));
	wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
	ESP_ERROR_CHECK(esp_wifi_init(&cfg));
	ESP_ERROR_CHECK(esp_wifi_set_storage(WIFI_STORAGE_RAM));
	#if CONFIGURE_AS_SOFT_AP

		ESP_ERROR_CHECK(tcpip_adapter_dhcps_stop(TCPIP_ADAPTER_IF_AP));
		memset(&info, 0, sizeof(info));
		IP4_ADDR(&info.ip, 192, 168, 1, 1);
		IP4_ADDR(&info.gw, 192, 168, 1, 1);
		IP4_ADDR(&info.netmask, 255, 255, 255, 0);
		ESP_ERROR_CHECK(tcpip_adapter_set_ip_info(TCPIP_ADAPTER_IF_AP, &info));
		ESP_ERROR_CHECK(tcpip_adapter_dhcps_start(TCPIP_ADAPTER_IF_AP));

		ESP_ERROR_CHECK( esp_wifi_set_ps( WIFI_PS_NONE ) );
		wifi_config_t wifi_config = 
		{
			.ap = 
			{
				.ssid = EXAMPLE_WIFI_SSID,
				.password = EXAMPLE_WIFI_PASS,
				.authmode = WIFI_AUTH_WPA2_PSK,
 				// .channel = 11,
 				.max_connection = 2,
			},
		};
		ESP_LOGI(TAG, "Setting WiFi configuration SSID %s...", wifi_config.ap.ssid);
		ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_AP));
		ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_AP, &wifi_config));
	#else
		wifi_config_t wifi_config = 
		{
			.sta = 
			{
				.ssid = EXAMPLE_WIFI_SSID,
				.password = EXAMPLE_WIFI_PASS,
			},
		};
		ESP_LOGI(TAG, "Setting WiFi configuration SSID %s...", wifi_config.sta.ssid);
		ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA));
		ESP_ERROR_CHECK(esp_wifi_set_config(ESP_IF_WIFI_STA, &wifi_config));
	#endif
	ESP_ERROR_CHECK(esp_wifi_start());
	// dns_server_start();   ---- TODO  FIXIT need to figure out how to make DNS hijack work
        (void) start_webserver();
}

void app_main()
{
    static httpd_handle_t server = NULL;

    ESP_ERROR_CHECK(nvs_flash_init());
    init_wifi_and_httpd(&server);
}

At the top of the file you'll see the #define CONFIGURE_AS_SOFT_AP. If you set this to 1, the ESP32 will establish a hotspot with the login credentials below. If you set it to 0, it will attempt to connect to your hotspot using whatever login credentials you set up with "make menuconfig"

With this very simple index.html, I'm able to catch and decode things posted with the dropdown menu. I've kinda been doing this whole thing from the bottom up, and my next step was to figure out how to catch and decode a big file, and if I could figure that out, then figure out what I need to do to write those file chunks to flash and then finally figure out how to get the bootloader to launch into that code at the next reboot instead of what's currently running (i.e. ping pong bootloader).

But when the incoming file gets to be a certain size, the http server crashes before I can even take a look at whats being sent. And in the course of testing this stripped down version, I just realized that when configured as a Soft AP, it can accept much larger files before it crashes. Earlier today I had been testing my actual app with the esp32 logging into my home network just because that's a lot faster and easier. When set up in station mode it chokes on pretty small files. So that's a little mystery. But in any event, there is a point where when you try to upload a large enough file, the http server barfs on the html header with this message:

W (47506) httpd_parse: parse_block: response uri/header too big
W (47506) httpd_txrx: httpd_resp_send_err: 500 Server Error - Server has encountered an unexpected error

Once that happens, only a hard reboot recovers.

OBDave · Postby **OBDave** » Thu May 30, 2019 2:00 am

Any suggestions, anyone?

Postby **ESP_Sprite** » Thu May 30, 2019 4:37 am

From what I remember, form-multipart uploads are a bit hard to parse in an embedded context as there needs to be lots of processing to get the actual data. It's easier to do this using a bit of javascript that fires off a 'raw' request. We have an example for that: see esp-idf/examples/protocols/http_server/file_serving/ . Perhhaps it's better to build off of that.

martinayotte · Postby **martinayotte** » Thu May 30, 2019 1:24 pm

@ESP_Sprite , I've used WebServer on ESP8266 in the past, which was providing multipart handling, and it was working fine.
Probably the same kind of code exist for ESP32 ...

OBDave · Postby **OBDave** » Mon Jun 03, 2019 6:43 pm

Been banging away at this for awhile and I found this post viewtopic.php?t=7685. No context is given but it seems that the max HTTP request header length had recently been increased from some unknown value to 512. This was a few years ago, and 512 is now the current default value. I increased it to 1024 and my crash issue went away.

I am now able to upload files of arbitrary size without the server crashing on me. So that's good. My next milestone will be figuring out how to actually catch the file at the server. That should be be a new post as I'm now drifting off topic, but just to close things out here, here's where I'm at now. I've never done any server-side HTML so this is all very new to me.

So I can select a file to upload, and my lil ESP32 server does not crash. I can call httpd_req_recv() repeatedly and seemingly receive the entire file, plus all the HTTP headers, but I don't see an obvious way to tell where the header ends and the file begins. I'm watching everything go by in wireshark, so I can see exactly what gets POSTed, but I don't see a foolproof way to separate the header from the actual payload, and since the payload will be an excusable binary image, I need to get this right every time.

Is there a library function to do this that I don't know about? And I'll submit this with more detail in a new thread.

simple standalone bootloader suitable for customers / end users?

simple standalone bootloader suitable for customers / end users?

Re: simple standalone bootloader suitable for customers / end users?

Re: simple standalone bootloader suitable for customers / end users?

Re: simple standalone bootloader suitable for customers / end users?

Re: simple standalone bootloader suitable for customers / end users?

Re: simple standalone bootloader suitable for customers / end users?

Re: simple standalone bootloader suitable for customers / end users?

Re: simple standalone bootloader suitable for customers / end users?

Re: simple standalone bootloader suitable for customers / end users?

Re: simple standalone bootloader suitable for customers / end users?

Who is online

About Us

Extra

Information