How to write a csv file along my .bin code into ESP32 4MB flash

[Hossein]

Hi
I need to put a .csv file along my code into ESP32 4MB flash in order to my application works properly. I don't want to hard code this data because different products need different .csv files. I placed .csv file on the main folder and then open it using fopen(). During building this program I got this error "initializer element is not constant". How should I solve this problem?

[ESP_Angus]

The main directory is just part of the source code used to build the app .bin, it doesn't get copied onto the device.

If you want the file to be part of the app .bin then you can embed it using the build system. This means you can access the data as an array or a string in your app (not as a file).

For ESP-IDF V4.0 (master branch):
https://docs.espressif.com/projects/esp ... inary-data

For ESP-IDF v3.x (stable releases, etc):
https://docs.espressif.com/projects/esp ... inary-data

If you specifically want to access it as a file, you can use SPIFFS or one of the other supported filesystems to flash a filesystem into the flash alongside the app .bin. But for a single read-only file this is probably not worth it.

[ESP_igrr]

Alternatively, if you want to use the same application binary for different products, but have them use different configuration CSV files, look into NVS partition generator utility (https://docs.espressif.com/projects/esp ... n_gen.html) and the mass manufacturing utility (https://docs.espressif.com/projects/esp ... s_mfg.html). These tools can convert CSV files in a predefined format into NVS partition images. You can flash the NVS partition image to the device, along with the application. The application can then access this configuration data using NVS APIs (https://docs.espressif.com/projects/esp ... flash.html).

(All links are for the development ("master") branch. To get the links for the latest release version, replace "latest" with "stable").

[Hossein]

Thanks ESP_Angus and ESP_igrr for your responses. What I mean by a CSV file is a dataset for our AI model that we want to write into flash and then use it during run time and it is a huge file compare to esp32 flash size , it is around 4MB and I have to reduce my dataset so provide some space for my program , so I don't think it is reasonable to use NVS flash because it is mentioned on Non-volatile storage library notes : "NVS works best for storing many small values, rather than a few large values of the type ‘string’ and ‘blob’. If you need to store large blobs or strings, consider using the facilities provided by the FAT filesystem on top of the wear levelling library.". By embedding file using build system I have my file on esp32 right now. The problem here is that I use a code like this to extract my file and put values on few arrays :

Code: Select all

extern const int8_t data_csv_start[] asm("_binary_data_csv_start");
extern const int8_t data_csv_end[] asm("_binary_data_csv_end");  
    int8_t a[10];
    int aa;
    int8_t b[10];
    int bb ;
    int8_t c[10];
    int cc ;
    int8_t d[10];
    int dd ;
    int8_t e[10];
    int ee;
	char buffer[128];
    for(int z = 0 ; z < 10 ; z++){
    	fgets(buffer,128,data_csv_start);
    	ESP_LOGE(TAG , "buffer : %s",buffer);
    	sscanf(buffer , "%d,%d,%d,%d,%d" , &aa , &bb , &cc , &dd , &ee);
    	a[z] = aa;
    	ESP_LOGE(TAG , "a[%d] = %d , b[%d] = %d , c[%d] = %d , d[%d] = %d , e[%d] = %d" , z , a[z] , z , b[z] , z , c[z] , z , d[z] , z , e[z]);
    }

But data_csv_start is incompatible pointer type for fgets. This code compile with this warning but when it runs on esp32 it throws following error : (xQueueGenericReceive)- assert failed!
abort() was called at PC 0x4008ce95 on core 0.
Could you please provide me a way to do this on esp32.

[ESP_igrr]

If you want to use fgets, then you first need to obtain a stream (FILE*) handle. To get a FILE* handle from a read-only string, use the following function:

FILE * stream = fmemopen((const char*) data_csv_start, data_csv_end - data_csv_start, "rb");

then you can use 'stream' as an argument of fgets.

[Hossein]

Thanks. Provided solutions solved my problem.