How to write a csv file along my .bin code into ESP32 4MB flash

Hossein
Posts: 4
Joined: Tue Aug 20, 2019 8:14 pm

How to write a csv file along my .bin code into ESP32 4MB flash

Postby Hossein » Tue Aug 20, 2019 8:48 pm

Hi
I need to put a .csv file along my code into ESP32 4MB flash in order to my application works properly. I don't want to hard code this data because different products need different .csv files. I placed .csv file on the main folder and then open it using fopen(). During building this program I got this error "initializer element is not constant". How should I solve this problem?

ESP_Angus
Posts: 2344
Joined: Sun May 08, 2016 4:11 am

Re: How to write a csv file along my .bin code into ESP32 4MB flash

Postby ESP_Angus » Wed Aug 21, 2019 12:04 am

The main directory is just part of the source code used to build the app .bin, it doesn't get copied onto the device.

If you want the file to be part of the app .bin then you can embed it using the build system. This means you can access the data as an array or a string in your app (not as a file).

For ESP-IDF V4.0 (master branch):
https://docs.espressif.com/projects/esp ... inary-data

For ESP-IDF v3.x (stable releases, etc):
https://docs.espressif.com/projects/esp ... inary-data

If you specifically want to access it as a file, you can use SPIFFS or one of the other supported filesystems to flash a filesystem into the flash alongside the app .bin. But for a single read-only file this is probably not worth it.

ESP_igrr
Posts: 2071
Joined: Tue Dec 01, 2015 8:37 am

Re: How to write a csv file along my .bin code into ESP32 4MB flash

Postby ESP_igrr » Wed Aug 21, 2019 10:13 am

Alternatively, if you want to use the same application binary for different products, but have them use different configuration CSV files, look into NVS partition generator utility (https://docs.espressif.com/projects/esp ... n_gen.html) and the mass manufacturing utility (https://docs.espressif.com/projects/esp ... s_mfg.html). These tools can convert CSV files in a predefined format into NVS partition images. You can flash the NVS partition image to the device, along with the application. The application can then access this configuration data using NVS APIs (https://docs.espressif.com/projects/esp ... flash.html).

(All links are for the development ("master") branch. To get the links for the latest release version, replace "latest" with "stable").

Hossein
Posts: 4
Joined: Tue Aug 20, 2019 8:14 pm

Re: How to write a csv file along my .bin code into ESP32 4MB flash

Postby Hossein » Thu Aug 22, 2019 12:39 am

Thanks ESP_Angus and ESP_igrr for your responses. What I mean by a CSV file is a dataset for our AI model that we want to write into flash and then use it during run time and it is a huge file compare to esp32 flash size , it is around 4MB and I have to reduce my dataset so provide some space for my program , so I don't think it is reasonable to use NVS flash because it is mentioned on Non-volatile storage library notes : "NVS works best for storing many small values, rather than a few large values of the type ‘string’ and ‘blob’. If you need to store large blobs or strings, consider using the facilities provided by the FAT filesystem on top of the wear levelling library.". By embedding file using build system I have my file on esp32 right now. The problem here is that I use a code like this to extract my file and put values on few arrays :

Code: Select all

extern const int8_t data_csv_start[] asm("_binary_data_csv_start");
extern const int8_t data_csv_end[] asm("_binary_data_csv_end");  
    int8_t a[10];
    int aa;
    int8_t b[10];
    int bb ;
    int8_t c[10];
    int cc ;
    int8_t d[10];
    int dd ;
    int8_t e[10];
    int ee;
	char buffer[128];
    for(int z = 0 ; z < 10 ; z++){
    	fgets(buffer,128,data_csv_start);
    	ESP_LOGE(TAG , "buffer : %s",buffer);
    	sscanf(buffer , "%d,%d,%d,%d,%d" , &aa , &bb , &cc , &dd , &ee);
    	a[z] = aa;
    	ESP_LOGE(TAG , "a[%d] = %d , b[%d] = %d , c[%d] = %d , d[%d] = %d , e[%d] = %d" , z , a[z] , z , b[z] , z , c[z] , z , d[z] , z , e[z]);
    }

But data_csv_start is incompatible pointer type for fgets. This code compile with this warning but when it runs on esp32 it throws following error : (xQueueGenericReceive)- assert failed!
abort() was called at PC 0x4008ce95 on core 0.
Could you please provide me a way to do this on esp32.

ESP_igrr
Posts: 2071
Joined: Tue Dec 01, 2015 8:37 am

Re: How to write a csv file along my .bin code into ESP32 4MB flash

Postby ESP_igrr » Thu Aug 22, 2019 7:36 am

If you want to use fgets, then you first need to obtain a stream (FILE*) handle. To get a FILE* handle from a read-only string, use the following function:

FILE * stream = fmemopen((const char*) data_csv_start, data_csv_end - data_csv_start, "rb");

then you can use 'stream' as an argument of fgets.

Hossein
Posts: 4
Joined: Tue Aug 20, 2019 8:14 pm

Re: How to write a csv file along my .bin code into ESP32 4MB flash

Postby Hossein » Tue Aug 27, 2019 12:15 am

Thanks. Provided solutions solved my problem.

Who is online

Users browsing this forum: Baidu [Spider] and 75 guests