loading AWS certs from SPIFFS

[Trialblazer47]

Hi so I was trying to load certificates from SPIFFS and had trouble. I thought it would be easy to simply flash certs and mount spiffs.

I modified the subcribe and publish code and edited sd Card section here is my code :

Code: Select all

#if defined(CONFIG_EXAMPLE_EMBEDDED_CERTS)
    mqttInitParams.pRootCALocation = (const char *)aws_root_ca_pem_start;
    mqttInitParams.pDeviceCertLocation = (const char *)certificate_pem_crt_start;
    mqttInitParams.pDevicePrivateKeyLocation = (const char *)private_pem_key_start;

#elif defined(CONFIG_EXAMPLE_FILESYSTEM_CERTS)
    mqttInitParams.pRootCALocation = ROOT_CA_PATH;
    mqttInitParams.pDeviceCertLocation = DEVICE_CERTIFICATE_PATH;
    mqttInitParams.pDevicePrivateKeyLocation = DEVICE_PRIVATE_KEY_PATH;
#endif

    mqttInitParams.mqttCommandTimeout_ms = 20000;
    mqttInitParams.tlsHandshakeTimeout_ms = 5000;
    mqttInitParams.isSSLHostnameVerify = true;
    mqttInitParams.disconnectHandler = disconnectCallbackHandler;
    mqttInitParams.disconnectHandlerData = NULL;

#ifdef CONFIG_EXAMPLE_SDCARD_CERTS
ESP_LOGI(TAG, "Initializing SPIFFS");

esp_vfs_spiffs_conf_t conf = {
.base_path = "/spiffs",
.partition_label = NULL,
.max_files = 5,
.format_if_mount_failed = false
};
// Use settings defined above to initialize and mount SPIFFS filesystem.
// Note: esp_vfs_spiffs_register is an all-in-one convenience function.
  esp_err_t ret = esp_vfs_spiffs_register(&conf);

  if (ret != ESP_OK) {
      if (ret == ESP_FAIL) {
          ESP_LOGE(TAG, "Failed to mount or format filesystem");
      } else if (ret == ESP_ERR_NOT_FOUND) {
          ESP_LOGE(TAG, "Failed to find SPIFFS partition");
      } else {
          ESP_LOGE(TAG, "Failed to initialize SPIFFS (%d)", ret);
      }
      return;
  }else
    ESP_LOGW(TAG,"spiffs register Status: %d",ret);

  size_t total = 0, used = 0;
  ret = esp_spiffs_info(NULL, &total, &used);
  if (ret != ESP_OK) {
      ESP_LOGE(TAG, "Failed to get SPIFFS partition information");
  } else {
      ESP_LOGW(TAG, "Partition size: total: %d, used: %d", total, used);
  }
  if( esp_spiffs_mounted(conf.partition_label))
  {
    ESP_LOGW(TAG,"SPIFFS Mounted.......");
  }else
    ESP_LOGW(TAG,"SPIFFS NOT Mounted.......");
  // Check if destination file exists
    struct stat st;
    if (stat(ROOT_CA_PATH, &st) == 0) {
      ESP_LOGI(TAG,"FILE EXIST")
    }
    if (stat(DEVICE_CERTIFICATE_PATH, &st) == 0) {
      ESP_LOGI(TAG,"FILE EXIST")
    }
    if (stat(DEVICE_PRIVATE_KEY_PATH, &st) == 0) {
      ESP_LOGI(TAG,"FILE EXIST")
    }
#endif

    rc = aws_iot_mqtt_init(&client, &mqttInitParams);

there is some problem that I face
my log

Code: Select all

I (276) subpub: AWS IoT SDK Version 2.2.1-
I (276) subpub: Initializing SPIFFS
W (296) subpub: spiffs register Status: 0
W (296) subpub: Partition size: total: 52961, used: 5773
W (296) subpub: SPIFFS Mounted.......
I (296) subpub: FILE EXIST
I (306) subpub: FILE EXIST
I (306) subpub: FILE EXIST
I (1606) wifi: n:11 0, o:1 0, ap:255 255, sta:11 0, prof:1
I (2596) wifi: state: init -> auth (b0)
I (3596) wifi: state: auth -> init (2)
I (3596) wifi: n:11 0, o:11 0, ap:255 255, sta:11 0, prof:1
I (3716) wifi: n:11 0, o:11 0, ap:255 255, sta:11 0, prof:1
I (3716) wifi: state: init -> auth (b0)
I (3716) wifi: state: auth -> assoc (0)
I (3726) wifi: state: assoc -> run (10)
I (3746) wifi: connected with UNICORN, channel 11
I (5106) event: sta ip: 192.168.43.197, mask: 255.255.255.0, gw: 192.168.43.1
I (5106) subpub: Connecting to AWS...
E (5106) aws_iot: failed!  mbedtls_x509_crt_parse returned -0x3e00 while parsing root cert
E (5116) subpub: Error(-19) connecting to :8883
E (6116) aws_iot: failed!  mbedtls_x509_crt_parse returned -0x3e00 while parsing root cert
E (6116) subpub: Error(-19) connecting to :8883

[Trialblazer47]

So I added code to check the file data is correct or not and this is what esp reads for all three files .

Code: Select all

I (306) subpub: FILE EXIST
I (306) subpub: Read from file: '???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????@0'???????????????????????????????????????????????????????????????????@?(
I (1606) wifi: n:11 0, o:1 0, ap:255 255, sta:11 0, prof:1

partition has data as I also get

Code: Select all

I (276) subpub: Initializing SPIFFS
W (286) subpub: spiffs register Status: 0
W (286) subpub: Partition size: total: 52961, used: 5773
W (296) subpub: SPIFFS Mounted.......
I (296) subpub: FILE EXIST
I (306) subpub: FILE EXIST
I (306) subpub: FILE EXIST

so is it that it's encrypted ? or something else ? I am using SPIFFS, mkspiffs tool esptool for flashing spiffs image.

[ESP_Angus]

Probably not encrypted, unless you're using flash encryption on the ESP32. Are the certs in PEM format? (if so they should be plaintext readable)

Can you please share the code you're using to print the certificate contents?

[Trialblazer47]

yes the spiffs.bin file has all data with some '?' in it. and content of certs is also easily visible.
this is the current code I am testing.

Code: Select all

  ret = esp_spiffs_info(NULL, &total, &used);
  if (ret != ESP_OK) {
      ESP_LOGE(TAG, "Failed to get SPIFFS partition information");
  } else {
      ESP_LOGW(TAG, "Partition size: total: %d, used: %d", total, used);
  }
  if( esp_spiffs_mounted(conf.partition_label))
  {
    ESP_LOGW(TAG,"SPIFFS Mounted.......");
  }else
    ESP_LOGW(TAG,"SPIFFS NOT Mounted.......");
  // Check if destination file exists before renaming
    struct stat st;
    if (stat(ROOT_CA_PATH, &st) == 0) {
      ESP_LOGI(TAG,"FILE EXIST");
    }
    if (stat(DEVICE_CERTIFICATE_PATH, &st) == 0) {
      ESP_LOGI(TAG,"FILE EXIST");

    }
    if (stat(DEVICE_PRIVATE_KEY_PATH, &st) == 0) {
      ESP_LOGI(TAG,"FILE EXIST");
    }
    // First create a file.
   ESP_LOGI(TAG, "Opening file");
   FILE* f = fopen("/spiffs/test.txt", "w");
   if (f == NULL) {
       ESP_LOGE(TAG, "Failed to open file for writing");
       return;
   }
   fprintf(f, "Hello World!\n");
   fclose(f);
   ESP_LOGI(TAG, "File written");
    //yes this works
    f = fopen("/spiffs/test.txt", "r");
    char line[1758];
    fgets(line, sizeof(line), f);
    if(f!=NULL){
      line[1758]=0;
      ESP_LOGI(TAG, "Read from file: '%s'", line);
    }else{
      ESP_LOGI(TAG,"Failed to open file");
    }
    fclose(f);
    //this does not
    //clearing array
    memset(line,0xFF,1757);
    f = fopen(DEVICE_PRIVATE_KEY_PATH, "r");
   fgets(line, sizeof(line), f);
   if(f!=NULL){
     line[1758]=0;
     ESP_LOGI(TAG, "Read from file: '%s'", line);
   }else{
     ESP_LOGI(TAG,"Failed to open file");
   }

if I create a file write it and read, it works but if I flash spiffs.bin it has all the files and it detects it but does not show correct file contents.

[ESP_Angus]

Can you test the result of fgets()? If it returns NULL then the read failed for some reason and you'll be left with a buffer full of 0xFF plus a null terminator. This would explain the '?'s.

Can you also please try changing your code to something like:

Code: Select all

memset(line, 0, sizeof(line));
int r = fread(line, 1, sizeof(line)-1, f);
ESP_LOGI(TAG, "read %d bytes: %s\n" , r, line);

Thanks,

Angus

[Trialblazer47]

So I tried this:

Code: Select all

    ESP_LOGI(TAG, "Reading file");
    f = fopen("/spiffs/aws-root-ca.pem", "r");
    if (f == NULL) {
        ESP_LOGE(TAG, "Failed to open file for reading");
        return;
    }
    char line[64];
    int r = fgets(line, sizeof(line), f);
    ESP_LOGI(TAG,"fgets Returned %d",r);
    char line2[64];
    memset(line2, 0, sizeof(line));
    r = fread(line2, 1, sizeof(line2)-1, f);
    ESP_LOGI(TAG, "read %d bytes: %s\n" , r, line2);
    fclose(f);

and Output:

Code: Select all

I (266) example: Initializing SPIFFS
I (276) example: Partition size: total: 52961, used: 6275
I (276) example: Reading file
I (276) example: fgets Returned 0
I (276) example: read 0 bytes: 

@0'286) example: Read from file: '????????P@4	
I (286) example: SPIFFS unmounted

[tele_player]

fgets() returns char*, not int.

[ESP_Angus]

Yes, the reads seem to indicate an empty file.

Are you sure the path is right? I noticed you used DEVICE_CERTIFICATE_PATH with stat and then hardcoded a path with fopen()

What's the value of st->st_size after you call stat() on each file? To avoid confusion, please update your code so 'st' is initialized correctly ie:

Code: Select all

  struct stat st = {};

[Trialblazer47]

yes, the path is right. I am actually now trying this with spiffs example file so I hardcoded it here and if it was wrong then fopen should not fail?

note: I am doing this with files mounted with help of esptool and image created with mkspiffs.
if you could try uploading file to spiffs partition and read it.

here is the code I changed :

Code: Select all

    struct stat st;
    if (stat("/spiffs/aws-root-ca.pem", &st) == 0) {
        ESP_LOGI(TAG,"FILE EXIST");
    }
    ESP_LOGI(TAG,"st->st_size: %d",(int)st.st_size);
    ESP_LOGI(TAG, "Reading file");
    f = fopen("/spiffs/aws-root-ca.pem", "r");
    if (f == NULL) {
        ESP_LOGE(TAG, "Failed to open file for reading");
        return;
    }
    
    char line[64];
    char *str = fgets(line, sizeof(line), f);
   // ESP_LOGI(TAG,"fgets Returned %s",str);
    
    char line2[64];
    memset(line2, 0, sizeof(line));
    int r = fread(line2, 1, sizeof(line2)-1, f);
    ESP_LOGI(TAG, "read %d bytes: %s\n" , r, line2);
    fclose(f);

OUTPUT:

Code: Select all

I (0) cpu_start: Starting scheduler on APP CPU.
I (267) example: Initializing SPIFFS
I (277) example: Partition size: total: 52961, used: 6275
I (277) example: FILE EXIST
I (277) example: st->st_size: 1760
I (277) example: Reading file
I (287) example: read 0 bytes: 

@0'287) example: Read from file: '????????????P@<	
I (297) example: SPIFFS unmounted

[Trialblazer47]

any help ?? or can someone make example.