Encrypted OTA + Flash Encryption + Secure Boot

[javqui]

I have been working with the secure boot and flash encryption and OTA for the last days.
All of them were working individually without issues (http and https OTA for about a year).
I have several days with a problem with the OTA update and flash encryption. I have a huge pressure to release this section, so my actions are touching the desperate field:
My First try fail. Any file downloaded usint ther OTA API (I was using the idf example) trigger an error at esp_ota_write (trying to write to an encrypted flash.)
My action: removed the encrypted part in the ota API code that handle the spi_flash_write_encrypted. Now I upload to the server an encrypted firmware file, and the OTA software write directly (without encryption) to an encrypted OTA partition.
This modification include the way that the encrypted otadata partition is updated. (it write like normal flash write, but the sector is not encrypted)

My problem: The problem is at the boot. the bootloader can't understand the otadata because it's not encrypted.
If I read manually the otadata partition, encrypt it (i have the keys to do espsecure.py encrypt_flash_data...) and flash back encrypted to the Flash, the bootloader recognize the data and switch to the proper OTA partition successfully. The downloaded encrypted firmware works without any issues. it works exactly as I need. Just the bootloader doesn't talk the same language than the modified OTA API regarding the otadata partition.

My options were fix the bootloader (remove encryption from otadata) or fix the ota API code when try to write to the encrypted otadata partition.
Because the first will introduce too many modification at the bootloader, I just continue with my second option.

Going a little further with the API code, the problem happen at the spi_flash_write_encrypted() routine, included in components/spi_flash/flash_ops.c

Code: [Select all] [Expand/Collapse]

1. esp_err_t IRAM_ATTR spi_flash_write_encrypted(size_t dest_addr, const void *src, size_t size)
2. {
3.     CHECK_WRITE_ADDRESS(dest_addr, size);
4.     const uint8_t *ssrc = (const uint8_t *)src;
5.     if ((dest_addr % 16) != 0) {
6.         return ESP_ERR_INVALID_ARG;
7.     }
8.     if ((size % 16) != 0) {
9.         return ESP_ERR_INVALID_SIZE;
10.     }
11.     COUNTER_START();
12.     esp_rom_spiflash_result_t rc;
13.     rc = spi_flash_unlock();               //<<<<< error I can't investigate further. only know that the result code is 0x6001
14.  
15.     if (rc == ESP_ROM_SPIFLASH_RESULT_OK) {
16.                                                                       // <<< never get here  
17.         /* esp_rom_spiflash_write_encrypted encrypts data in RAM as it writes,
18.            so copy to a temporary buffer - 32 bytes at a time.
19.  
20.            Each call to esp_rom_spiflash_write_encrypted takes a 32 byte "row" of
21.            data to encrypt, and each row is two 16 byte AES blocks
22.            that share a key (as derived from flash address).
23.         */
24.         uint8_t encrypt_buf[32] __attribute__((aligned(4)));
25.         uint32_t row_size;
26. ......

GeSHi © Codebox Plus Extension

I added a loop to try a few times with a 100ms delay separation. The routine always return the same error: 0x6001.

I will really appreciate any help on this topic.
Only this detail (maybe is more than a detail) is pending to make "Encrypted OTA" + "secure boot" + "flash encrypted" work


Thanks in advance for any light.

[WiFive]

What is your hardware?
What was your modification?

[ESP_Angus]

Hi javqui,

Thanks for the detailed report. I have a couple more questions:

- Exactly which version of ESP-IDF are you using?

- Did the encrypted OTA updates previously work OK? If yes, was this with the same ESP-IDF version, or a different version? Has anything else changed in the configuration?

- What hardware do you have?


Angus

[javqui]

Hi, Thanks for so quick response.

I forgot to mention some details, I was too tired yesterday.
Here are more details:
Hardware: ESP32-WROVER-IB (16MB)
dev v4.0-dev-1018-gbbafd5066

I went a little further yesterday and remove "temporarily" the lock/unlock SPI flash check. I know that is not the final solution, but it works perfect (few rounds between different ota partitions successfully). Ota data partition and is working encrypted now.

The board is able to download a full authenticated and encrypted firmware from a server using HTTPS (http also). It requires 3 different crypto keys, (bootloader/firmware authentication, firmware flash encryption and https key). The https is optional, but why not used if the espress-if developers release and it works perfectly. (Thanks guys)

I was unable to debug with the JTAG on this particular board, so it was a little difficult, sorry to not provide extra details. (suggestion: don't burn the JTAG efuse so early, to test this feature)

Looks like the issue is related with the SPI flash driver on this hardware. the far as I went yesterday was to the routine "esp_flash_set_chip_write_protect" where the write encryption fail.

Because I found a temporal solution to write encrypted data on this particular spi flash hardware (the solution was just to continue even if the unlock/lock return an error), I need to move forward with other features this week, and return back later to this topic again.

Additional observations (hope will be useful for somebody)
I read a comment on this forum about why to use encrypted firmware for OTA, some people thing that HTTPS is enough.
On this case, a potential security hole is on the remote http server provider itself, we should try to avoid upload plain firmware updates on a remote server.

Javqui.-

[ESP_Angus]

Hi Javqui,

Thanks for the additional details. There have been some changes in the SPI flash during IDF V4.0 development, so this may be a regression. We will debug it and get back to you ASAP.

(For a more stable but still prerelease experience you could try the release/v3.3 branch. See https://docs.espressif.com/projects/esp ... sions.html for full details.)


Regarding security issues serving plaintext images from an untrusted server, we're aware this is a problem for a lot of users. At the moment the best option I can recommend is to encrypt the image using some other encryption scheme, store the shared encryption key inside the firmware, and decrypt on the device and then write with flash encryption during OTA. The advantage to this is not needing to record the flash encryption key for each device.

Long term, we are planning to add a feature to support this (pre-encrypted binary images and OTA) without needing additional code. But we don't have an ETA for that.

Angus

[javqui]

Thanks so much ESP_Angus for your suggestions.

Maybe I'm missing something here.
What I did was: encrypt the firmware with the same key stored in flash (BLOCK1), upload to the remote server, then the ESP32 will download it using the OTA API modified. The only modification was to write to flash "as is", just as plain write without encryption, because data it's already encrypted.
It removes the buffer requirements (Alignment 16) and the online encryption.
From the OTA download perspective, there is no difference between encrypted or not encrypted data, only writing to the OTAdata partition will be encrypted.
A simple readback will decrypt and confirm that the downloaded data satisfy basic conditions (partition header).

Did you see any security problem with this approach?

Thanks

[WiFive]

Only problem is having non-unique flash encryption keys per device.

[ESP_Angus]

That's right, as WiFive mentions we don't recommend this as we don't recommend using the same key on all ESP32s.

The reason being that if the key is leaked or recovered from the hardware by some current unforeseeen mechanism, this effects all devices rather than just one device.

Also it may be possible for an attacker to transplant data between devices (for example, save target Wi-Fi credentials on Device A which they control and then migrate this encrypted data onto Device B).

Using a separate encryption scheme for OTA also has some similar potential problems (if the key is leaked then all OTA images can be decrypted). However this is only the OTA .bin image and not, for example, user data or key data stored on the device which could be recovered or manipulated.

Depending on your security threat model these things may or may not be a concern for you.

[javqui]

Thanks,
Great observations. I was balancing the pros and cons about an unique key per device.

[hereiam]

Hi Javqui,

Thanks for the additional details. There have been some changes in the SPI flash during IDF V4.0 development, so this may be a regression. We will debug it and get back to you ASAP.

(For a more stable but still prerelease experience you could try the release/v3.3 branch. See https://docs.espressif.com/projects/esp ... sions.html for full details.)


Regarding security issues serving plaintext images from an untrusted server, we're aware this is a problem for a lot of users. At the moment the best option I can recommend is to encrypt the image using some other encryption scheme, store the shared encryption key inside the firmware, and decrypt on the device and then write with flash encryption during OTA. The advantage to this is not needing to record the flash encryption key for each device.

Long term, we are planning to add a feature to support this (pre-encrypted binary images and OTA) without needing additional code. But we don't have an ETA for that.

Angus

Hello Angus,

can you me if we can now update by OTA with the encrypted firmware ?

thanks