ESP32 as ISP

hgptamn
Posts: 26
Joined: Mon Oct 16, 2017 4:47 pm

Re: ESP32 as ISP

Postby hgptamn » Thu May 24, 2018 10:33 am

I finally managed to program the external flash using the ESP32 core for Arduino IDE and two more components:
1. SPIFFS: https://github.com/espressif/arduino-es ... ies/SPIFFS
2. A cool library developed by Marzogh: https://github.com/Marzogh/SPIMemory

For uploading the 3 binaries (bootloader.bin (0x1000) , partitions_singleapp.bin (0x8000), my_app.bin (0x10000)) into SPIFFS I've used the following tool: https://github.com/me-no-dev/arduino-esp32fs-plugin
You can also create a custom partition table and have the SPIFFS VFS as big as you want. More info over here: https://desire.giesecke.tk/index.php/20 ... duino-ide/
Once the partition table is modified, esp32fs-plugin for Arduino IDE will acknowledge this at the next upload.

You can also use this other tool developed by loboris in order to prepare a SPIFFS image and upload it to the flash memory of the master: https://github.com/loboris/ESP32_spiffs_example
Unfortunately, Arduino IDE SPIFFS doesn't see this partion, but ESP-IDF SPIFFS does. So if you decide on using this tool for uploading the SPIFFS image to the master, you need to use ESP-IDF in order to have access to the files that were uploaded. I guess you could also use Arduino IDE SPIFFS and FS.h with this tool if you modify the partitions in

Code: Select all

./Arduino/hardware/espressif/esp32/tools/partitions
I will try my best to describe the process. In my case the slave board got connected to the master by pogo pins, hence the detection of the board being connected to the master was done by continuously looking for JEDEC ID of the slave in the flash_detected() method. This required that the CHIP_PU line was LOW when the board got connected to the master.

The entire process is as following:
1. Prior to connecting the external flash to the master board, upload the 3 binaries listed above to the programmer board flash memory (will call this board master) using SPIFFS. You can either use esp32fs-plugin and Arduino IDE or loboris's tool to do so.
2. Connect the external flash to the master as shown in the image posted in post #1 of this topic
3. Disable the board to be programmed (will call it slave) by putting CHIP_PU LOW. This ensures that the slave's flash is free to be accessed by the master on the SPI bus.
4. Upload the 3 binaries to their corresponding address:
bootloader.bin (0x1000) , partitions_singleapp.bin (0x8000), my_app.bin (0x10000)
You can use ESP32 Download Tool prior to this to make sure that the binaries do work as expected. Demo on how to use the tool can be found over here.
5. Once the binaries got uploaded enable the slave by putting CHIP_PU HIGH. The board should now boot with the new firmware uploaded at step 4.

The Arduino IDE code I've used to implement the above process is the following:

Code: Select all

#include <SPI.h>
#include "FS.h"
#include "SPIFFS.h"


// Highest page number is 0xffff=65535
int page_number = 0xFFFF;
unsigned char w_page[256];
unsigned char r_page[256];

unsigned char dummy_wbuf[4096];
unsigned char dummy_wbuf2[4096];

//SPI Flash constants
#define SPISPEED 20000000
#define CSPIN  15
#define CPU 25
#define MISO 12
#define MOSI 13
#define SCK 14

//EN pin for logic level converter in case you are using one
//The flash with which I was working (GD25LQ64C) was a 1.8V one.
//Logic level shifting between ESP32 (3.3V) and External Flash(1.8V) was nedded
//Ignore if you don't use a logic level shifter between the external flash and the main uC
#define CONVERTER 27 

#define STAT_WIP 1
#define STAT_WEL 2

#define CMD_WRITE_STATUS_REG   0x01
#define CMD_PAGE_PROGRAM       0x02
#define CMD_READ_DATA          0x03
#define CMD_WRITE_DISABLE      0x04//not tested
#define CMD_READ_STATUS_REG1   0x05
#define CMD_READ_STATUS_REG2   0x35
#define CMD_WRITE_ENABLE       0x06
#define CMD_READ_HIGH_SPEED    0x0B//not tested
#define CMD_SECTOR_ERASE       0x20//not tested
#define CMD_BLOCK32K_ERASE     0x52//not tested
#define CMD_RESET_DEVICE       0xF0//<<-different from winbond
#define CMD_READ_ID            0x9F
#define CMD_RELEASE_POWER_DOWN 0xAB//not tested
#define CMD_POWER_DOWN         0xB9//not tested
#define CMD_CHIP_ERASE         0x60
#define CMD_BLOCK64K_ERASE     0xD8//not tested

unsigned char flash_wait_for_write = 0;
unsigned long start_time, total_time;

#define BUFFERSIZE 4096


unsigned char flash_get_status(void) {
  unsigned char c;

  digitalWrite(CSPIN, LOW);

  //Read S7-S0
  SPI.transfer(CMD_READ_STATUS_REG1);
  c = SPI.transfer(0x00);

  digitalWrite(CSPIN, HIGH);

  return c;
}

void write_pause(void)
{
  if (flash_wait_for_write) {
    do {
    } while (flash_get_status() & STAT_WIP);

    flash_wait_for_write = 0;
  }
}

void flash_read_id(uint16_t id[])
{
  write_pause();

  //set control register
  SPI.beginTransaction(SPISettings(SPISPEED, MSBFIRST, SPI_MODE0));
  digitalWrite(CSPIN, LOW);
  SPI.transfer(0x9F);
  id[0] = SPI.transfer(0);
  id[1] = SPI.transfer(0);
  id[2] = SPI.transfer(0);
  digitalWrite(CSPIN, HIGH);
  SPI.endTransaction();
}

bool flash_detected() {
  uint16_t id[3] = {0};
  bool detected = false;
  start_time = millis();

  Serial.println("Awiting target to be inserted");

  do {
    flash_read_id(id);
    if (id[0] != 0x00 && id[0] != 0xFF) {
      Serial.println("\nFlash detected!");
      Serial.print("Flsh ID: ");
      Serial.print(id[0], HEX); Serial.print(" "); Serial.print(id[1], HEX);  Serial.print(" ");  Serial.println(id[2], HEX);
      detected = true;
    }
    if (millis() - start_time >= 2000) {
      Serial.print(".");
      start_time = millis();
    }
    delay(100);
  } while (!detected);

  return detected;
}

void flash_page_program(unsigned char *wbuf, unsigned long address)
{

  write_pause();
  // Send Write Enable command
  digitalWrite(CSPIN, LOW);
  SPI.transfer(CMD_WRITE_ENABLE);
  digitalWrite(CSPIN, HIGH);

  SPI.beginTransaction(SPISettings(SPISPEED, MSBFIRST, SPI_MODE0));
  digitalWrite(CSPIN, LOW);
  SPI.transfer(CMD_PAGE_PROGRAM);
  // Send the 3 byte address
  SPI.transfer((address >> 16) & 0xFF);
  SPI.transfer((address >> 8) & 0xFF);
  SPI.transfer(address & 0xFF);

  // Now write 256 bytes to the page
  for (uint16_t i = 0; i < 256; i++) {
    SPI.transfer(*wbuf++);
  }

  digitalWrite(CSPIN, HIGH);
  SPI.endTransaction();
  
  // Indicate that next I/O must wait for this write to finish
  flash_wait_for_write = 1;
}

void listDir(fs::FS &fs, const char * dirname, uint8_t levels) {
  Serial.printf("Listing directory: %s\r\n", dirname);

  File root = fs.open(dirname);
  if (!root) {
    Serial.println("- failed to open directory");
    return;
  }
  if (!root.isDirectory()) {
    Serial.println(" - not a directory");
    return;
  }

  File file = root.openNextFile();
  while (file) {
    if (file.isDirectory()) {
      Serial.print("  DIR : ");
      Serial.println(file.name());
      if (levels) {
        listDir(fs, file.name(), levels - 1);
      }
    } else {
      Serial.print("  FILE: ");
      Serial.print(file.name());
      Serial.print("\tSIZE: ");
      Serial.println(file.size());
    }
    file = root.openNextFile();
  }
}

void flash_erase_sector_at_address(unsigned long address)
{

  write_pause();
  
  // Send Write Enable command
  digitalWrite(CSPIN, LOW);
  SPI.transfer(CMD_WRITE_ENABLE);
  digitalWrite(CSPIN, HIGH);

  digitalWrite(CSPIN, LOW);
  SPI.transfer(CMD_SECTOR_ERASE);
  
  SPI.transfer((address >> 16) & 0xff);
  SPI.transfer((address >> 8) & 0xff);
  SPI.transfer(address & 0xff);
  digitalWrite(CSPIN, HIGH);
  
  // Indicate that next I/O must wait for this write to finish
  flash_wait_for_write = 1;
}


void flash_read_sector_at_address(unsigned char *p, unsigned long address) {
  unsigned char *rp = p;
  int pages_per_sector = 16;

  write_pause();
  
  digitalWrite(CSPIN, LOW);
  SPI.transfer(CMD_READ_DATA);
  
  // Send the 3 byte address
  SPI.transfer((address >> 16) & 0xFF);
  SPI.transfer((address >> 8) & 0xFF);
  SPI.transfer(address & 0xFF);
  
  // Now read the sectors's data bytes (16 pages * 256 byetes/sector)
  for (unsigned long i = 0; i < pages_per_sector * 256; i++) {
    *rp++ = SPI.transfer(0);
  }
  digitalWrite(CSPIN, HIGH);
}

void flash_print_sector(unsigned char *rbuf, unsigned long address) {
  int sector_sz = 4096;
  for (int j = 0; j < sector_sz; j += 16)
  {
    printf("%08X  "
           "%02X %02X %02X %02X "
           "%02X %02X %02X %02X "
           "%02X %02X %02X %02X "
           "%02X %02X %02X %02X ",
           address + j,
           rbuf[j + 0], rbuf[j + 1], rbuf[j + 2], rbuf[j + 3],
           rbuf[j + 4], rbuf[j + 5], rbuf[j + 6], rbuf[j + 7],
           rbuf[j + 8], rbuf[j + 9], rbuf[j + 10], rbuf[j + 11],
           rbuf[j + 12], rbuf[j + 13], rbuf[j + 14], rbuf[j + 15]);
    for (int k = j; k < j + 16; k++)
    {
      printf("%c", isprint(rbuf[k]) ? rbuf[k] : '.');
    }
    printf("\n");
  }
}

void flash_read_block_at_address(unsigned long address) {
  int address_step = 0x1000;
  int sectors_to_read = 16;
  unsigned char rbuf[4096];
  unsigned long i;

  for (i = address; i < (address + ((sectors_to_read - 1) * address_step)); i += address_step) {
    printf("addr = %d \n", i);
    flash_read_sector_at_address(rbuf, i);
    flash_print_sector(rbuf, i);
    printf("\n");
  }
}


void flash_binary(const char * path, unsigned long address) {
  int indexCounter = 0;
  int addr_step = 0x1000;
  int sector_size = 4096;
  int page_step = 256; //each page has 256 bytes
  unsigned long offset = 0;
  uint8_t *rbuf = (uint8_t *) malloc(BUFFERSIZE);
  uint8_t *verify = (uint8_t *) malloc(BUFFERSIZE);
  if (rbuf == NULL) Serial.println("Buffer is NULL");
  if (verify == NULL) Serial.println("Buffer is NULL");


  unsigned char temp_buf[page_step] = {0};

  for (int k = 0; k < 4096; k++) rbuf[k] = k;

  Serial.printf("\n\nOpening file %s from SPIFFS", path);
  File file = SPIFFS.open(path);
  if (!file || file.isDirectory()) {
    Serial.println("- failed to open file for reading");
    return;
  }
  else {
    Serial.print("\nFile opened. Proceding to writing binaries");
  }

  unsigned long fileSize = file.size();
  int chunkSize = 4096;
  unsigned long numberOfChunks = (fileSize / chunkSize) + 1;
  unsigned long count = 0;
  unsigned long remainingChunks = fileSize;

  //move pointer to begging of file
  file.seek(0);

  //proceed to writing the file into flash one chunk at a time
  for (unsigned long i = 1; i <= numberOfChunks; i++) {

    if (remainingChunks - chunkSize < 0) {
      chunkSize = remainingChunks;
    }

    if (i == numberOfChunks)
      file.read(rbuf, fileSize % 4096);
    else
      file.read(rbuf, 4096);

    remainingChunks = remainingChunks - chunkSize;

    //erase the sector before writing to it
    flash_erase_sector_at_address(address);

    uint8_t *rbuf_address = rbuf;

    for (unsigned long q = address, offset = 0; q < address + sector_size; q += page_step, offset += 256) {
      rbuf_address = rbuf + offset;
      flash_page_program(rbuf_address, q);
    }

    //UNCOMMENT THE FOLLOWING SECTGION IN ORDER TO VERIFY DATA WAS WRITTEN CORECTLY
    /*
      flash_read_sector_at_address(verify, address);

      //verify that data was written corectly
      for (int i = 0; i < BUFFERSIZE; i++)
      if (rbuf[i] != verify[i])
        Serial.printf("\nWritten data corrupted at byte offset %d", i);
    */

    //jump to the next sector
    address += sector_size;
    indexCounter = 0;

  }

  //close file
  file.close();
}

void reset_target() {
  digitalWrite(CPU, LOW);
  delay(10);
  digitalWrite(CPU, HIGH);
  delay(10);
}

void setup() {

  int k = 0;

  Serial.begin(2000000);
  delay(10);
  Serial.println("Serial enabled!");

  pinMode(CONVERTER, OUTPUT);
 
  pinMode(CPU, OUTPUT);
  pinMode(CSPIN, OUTPUT);
  pinMode(MISO, INPUT);
  pinMode(MOSI, OUTPUT);
  pinMode(SCK, OUTPUT);


  digitalWrite(CSPIN, LOW);
  digitalWrite(MISO, LOW);
  digitalWrite(MOSI, LOW);
  digitalWrite(SCK, LOW);
  digitalWrite(CPU, LOW);

  SPI.begin(SCK, MISO, MOSI, CSPIN); // sck, miso, mosi, ss (ss can be any GPIO)

  if (!SPIFFS.begin(true)) {
    Serial.println("\nSPIFFS Mount Failed");
    return;
  }
  else {
    Serial.println("\nSPIFFS Mounted Successfuly");
    Serial.println("\nSPIFFS content");
    listDir(SPIFFS, "/", 0);
  }
}

void loop() {


  //disable target
  digitalWrite(CPU, LOW);
  digitalWrite(CONVERTER, HIGH);

  if (flash_detected()) {

    Serial.println("\nEnabling target");
    delay(1000);
    digitalWrite(CONVERTER, LOW);
    reset_target();
    delay(10);

    start_time = millis();
    total_time = millis();

    Serial.println("\nFlash programming started. DO NOT REMOVE THE TARGET!");
    delay(1000);

    //set led to programming color (yellow color)
    pixels.setPixelColor(NUMPIXELS - 1, pixels.Color(255, 125, 125));
    pixels.show();

    flash_binary("/bootloader.bin", 0x1000);
    Serial.printf("\nBootloader written in %u ms", millis() - start_time);

    start_time = millis();
    flash_binary("/partitions_singleapp.bin", 0x8000);
    Serial.printf("\nPartitions written in %u ms", millis() - start_time);

    start_time = millis();
    flash_binary("/test.bin", 0x10000);
    Serial.printf("\nFirmware written in %u ms", millis() - start_time);

    pixels.setPixelColor(NUMPIXELS - 1, pixels.Color(0, 255, 0));
    pixels.show();

    Serial.printf("\n Total time: %u ms", millis() - total_time);

    Serial.println("\nEnabling target");
    digitalWrite(CONVERTER, LOW);
    reset_target();

  }
  delay(30000);
}
Unfortunately it comes with the limitation of not being able to use QuadSPI for faster data transfers. The ESP32 Arduino Core doesn't support that by now.
I'm pretty sure it can also be done in ESP-IDF. However, the lack of SPI driver examples (there's only one posted over here) and the complexity of IDF got me turning to the ESP32 Arduino Core.
One good example, pointed out by @Veder_Mester, which may help some of you can be found over here: https://esp32.com/viewtopic.php?f=17&t=4666
Don't expect it to be easy to understand and follow though.

If someone could come with and IDF implementation of the above code that would be great! If it offered the possibility of QuadSPI that would be awesome!

Who is online

Users browsing this forum: Baidu [Spider] and 65 guests