Write RAM using bootloader serial protocol

[JayLogue]

inject a bundle of data into RAM

this has to be platform specific on the tool side

stores in the appropriate way for the particular platform

this has to be platform specific on the device side

So is it really just sleight of hand to force the process to appear consistent across platforms?

Given inevitable differences in hardware, I suppose all cross-platform portability is just "sleight of hand" at some level.

In this case, however, it is still valuable that many aspects of the implementation are common, even if some aren't. For example, the code for encoding and decoding the data bundle is the same, and leverages existing code, on both the host and device, that serves other purposes as well. Similarly, the code on the device that writes the data to persistent storage uses a cross-platform key-value storage abstraction that already exists for each platform and is used in the normal operation of the device.

Effectively, the only significant platform-dependent code that is unique to the provisioning process is the code that injects the data bundle into RAM and reboots the device. On the Nordic nRF52840, this is implemented in a few dozen lines of python code and a call out to a single Nordic-supplied command line tool. If I can figure out the reboot issue, the implementation for the ESP32 will be nearly as simple.

[WiFive]

I am just devil's advocate and elegant solutions are nice, but I don't think it's worth it versus just provisioning the key value store directly since the tools already exist and you are stuck in download mode.

You might be able to jump directly to the instruction for flash boot, assuming you can find it and it won't break stuff...

[JayLogue]

SUCCESS!

I resolved the restart issue by creating a small program to initiate a system reboot using the RTC watchdog:

Code: Select all

#include "esp_attr.h"
#include "soc/cpu.h"
#include "soc/soc.h"
#include "soc/rtc.h"
#include "soc/rtc_cntl_reg.h"

void main()
{
    // Fire the RTC watchdog after about a second.
    REG_SET_FIELD(RTC_CNTL_CLK_CONF_REG, RTC_CNTL_ANA_CLK_RTC_SEL, RTC_SLOW_FREQ_RTC);
    REG_WRITE(RTC_CNTL_WDTWPROTECT_REG, RTC_CNTL_WDT_WKEY_VALUE);
    REG_WRITE(RTC_CNTL_WDTCONFIG0_REG,
              RTC_CNTL_WDT_EN |
              RTC_CNTL_WDT_FLASHBOOT_MOD_EN_M |
              (RTC_WDT_STG_SEL_RESET_RTC << RTC_CNTL_WDT_STG0_S) |
              (1 << RTC_CNTL_WDT_SYS_RESET_LENGTH_S));
    REG_WRITE(RTC_CNTL_WDTCONFIG1_REG, 150000);
    REG_WRITE(RTC_CNTL_WDTFEED_REG, (1 << RTC_CNTL_WDT_FEED_S));
    
    while (1);
}

I then combined the raw code from the reboot program with the data I wished to inject into a single ESP32 firmware image using a small python script.

Code: Select all

#!/usr/bin/env python3

import os 
import sys
import struct
from functools import reduce

ESP_IMAGE_MAGIC = 0xe9

ESP_CHECKSUM_MAGIC = 0xef

# Data bundle to be injected (dummy value for testing purposes)
dataBundle = b'Hello'
dataBundleLoadAddr = 0x3FFE0000 - 0x100

# Small ESP32 program to initiate reboot of device after data loaded.
rebootProg = bytearray([
    0x70, 0x80, 0xf4, 0x3f, 0xff, 0xff, 0xff, 0x3f, 0xa4, 0x80, 0xf4, 0x3f, 0xa1, 0x3a, 0xd8, 0x50,
    0x8c, 0x80, 0xf4, 0x3f, 0x00, 0x0c, 0x00, 0xc0, 0x90, 0x80, 0xf4, 0x3f, 0xf0, 0x49, 0x02, 0x00,
    0xa0, 0x80, 0xf4, 0x3f, 0x00, 0x00, 0x00, 0x80, 0x36, 0x41, 0x00, 0x91, 0xf5, 0xff, 0x81, 0xf5,
    0xff, 0xc0, 0x20, 0x00, 0xa8, 0x09, 0x80, 0x8a, 0x10, 0xc0, 0x20, 0x00, 0x89, 0x09, 0x91, 0xf3,
    0xff, 0x81, 0xf1, 0xff, 0xc0, 0x20, 0x00, 0x99, 0x08, 0x91, 0xf2, 0xff, 0x81, 0xf1, 0xff, 0xc0,
    0x20, 0x00, 0x99, 0x08, 0x91, 0xf2, 0xff, 0x81, 0xf0, 0xff, 0xc0, 0x20, 0x00, 0x99, 0x08, 0x91,
    0xf1, 0xff, 0x81, 0xef, 0xff, 0xc0, 0x20, 0x00, 0x99, 0x08, 0x06, 0xff, 0xff,
])
rebootProgLoadAddr = 0x40080000
rebootProgEntryPoint = 0x40080028

numSegments = 2

# Encode the image header
# (magic, # segments, flash mode, flash size/freq, entry point)
encodedImage = struct.pack('<BBBBI', ESP_IMAGE_MAGIC, numSegments, 0, 0, rebootProgEntryPoint)

# Encode all zeros for the ESP32 extended header
encodedImage += b'\0' * 16

# Encode an image section containing the reboot program.
encodedImage += struct.pack('<II', rebootProgLoadAddr, len(rebootProg))
encodedImage += rebootProg

# Encode an image section containing the data bundle.
encodedImage += struct.pack('<II', dataBundleLoadAddr, len(dataBundle))
encodedImage += dataBundle

# Compute the checksum
toBeChecksummed = rebootProg + dataBundle
checksum = reduce(lambda a,b: a^b, toBeChecksummed, ESP_CHECKSUM_MAGIC)

# Encode trailing padding and checksum byte
encodedImage += b'\0' * (15 - (len(encodedImage) % 16))
encodedImage += struct.pack('B', checksum)

# Write the result to stdout
os.write(sys.stdout.fileno(), encodedImage)

With this, I was able to load the firmware image using the esptool load_ram command, observe the device reboot and subsequently see the main firmware successfully locate the injected data.

Code: Select all

$ python3 ./make-esp32-image.py > data-bundle-image
$ /home/jay/tools/esp-idf/components/esptool_py/esptool/esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 115200 --trace load_ram data-bundle-image && picocom --baud 115200 /dev/ttyUSB0

This produced the following output when the device rebooted:

Code: Select all

ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:5636
load:0x40078000,len:13860
entry 0x40078fd0
I (26) boot: ESP-IDF v3.0.7-65-gd0c373039 2nd stage bootloader
I (26) boot: compile time 18:20:37
I (26) boot: Enabling RNG early entropy source...
I (32) boot: SPI Speed      : 40MHz
I (36) boot: SPI Mode       : DIO
I (40) boot: SPI Flash Size : 4MB
I (44) boot: Partition Table:
I (48) boot: ## Label            Usage          Type ST Offset   Length
I (55) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (63) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (70) boot:  2 factory          factory app      00 00 00010000 00200000
I (78) boot: End of partition table
I (82) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x32738 (206648) map
I (163) esp_image: segment 1: paddr=0x00042760 vaddr=0x3ffc0000 size=0x03f50 ( 16208) load
I (170) esp_image: segment 2: paddr=0x000466b8 vaddr=0x40080000 size=0x00400 (  1024) load
I (170) esp_image: segment 3: paddr=0x00046ac0 vaddr=0x40080400 size=0x09550 ( 38224) load
I (195) esp_image: segment 4: paddr=0x00050018 vaddr=0x400d0018 size=0xe64d4 (943316) map
I (525) esp_image: segment 5: paddr=0x001364f4 vaddr=0x40089950 size=0x0519c ( 20892) load
I (543) boot: Loaded app from partition at offset 0x10000
I (543) boot: Disabling RNG early entropy source...
I (544) cpu_start: Pro cpu up.
I (547) cpu_start: Starting app cpu, entry point is 0x40080f20
I (0) cpu_start: App cpu up.
I (558) heap_init: Initializing. RAM available for dynamic allocation:
I (565) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM
I (571) heap_init: At 3FFD37C0 len 0000C840 (50 KiB): DRAM
I (577) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (583) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (589) heap_init: At 4008EAEC len 00011514 (69 KiB): IRAM
I (596) cpu_start: Pro cpu start user code
I (280) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
E (303) openweave-demo: Found Hello at 0x3ffdff00
E (303) openweave-demo: Found Hello at 0x3ffdff00
E (303) openweave-demo: Found Hello at 0x3ffdff00
E (303) openweave-demo: Found Hello at 0x3ffdff00
E (303) openweave-demo: Found Hello at 0x3ffdff00

[WiFive]

Nice work. Hope the ram blob has hash/checksum just in case.

[JayLogue]

Thanks. Final code will have a SHA256 integrity check.

[WiFive]

So is openweave going to become the homekit of Android or what?

[JayLogue]

Weave is the technology that underlies most of Nest’s products. OpenWeave is an open-source implementation of Weave that grew out of Nest’s internal codebase. In a nutshell, OpenWeave is an application-level communication framework for building IoT ecosystems. It’s highly portable, and runs on all categories of devices, from sensors with only 64K of RAM to cameras with many megabytes. It also runs on mobile devices, both iOS and Android, and some portions of it run in Nest’s cloud servers.

HomeKit occupies a similar space but is more narrow in scope.

[WiFive]

It is very cool! But to see wide adoption it is going to need to fill a homekit-like niche with
First class support on Android (official app)
Common device classes
Secure local pairing/onboading using PoP
Default to P2P/D2D communication whenever possible
Any Android device or Google hub device can be a WAN/cloud gateway (like homekit through iPad/aTV)
Local integration with Google assistant (no cloud)
Easy and free self-certification (so things like homebridge don't need to dominate)
Permissive use of trademarks/logos (so it becomes a household name)

Nest started as a very profitable hardware company, now it needs to become a free Android feature .

The hardware should be relaunched as Google Thermostat (works with Nest)