Initializing NVS on a large partition

[Oromis]

Hi,

our IoT product usually sends a steady stream of data to our server. If we lose connectivity to the server, we buffer the messages in NVS. To be able to bridge longer offline periods without any data loss, we want to use a large partition for NVS, which is why we upgraded our flash memory to 16 MB.

Unfortunately, I noticed that when I increase the NVS partition size, the chip won't start any more due to an out-of-memory error. Looking into the nvs_pagemanager.cpp file, it seams that the NVS system tries to allocate RAM for each page, which exhausts the heap. It works for a 1MB NVS partition, but fails for 2MB or larger partitions.

The backtrace (for a 6MB NVS partition):

Code: Select all

0x400d0b4f: __cxa_end_catch at /home/david/esp/esp-idf/components/cxx/./cxx_exception_stubs.cpp:13
0x40089273: invoke_abort at /home/david/esp/esp-idf/components/esp32/./panic.c:572
0x4008929f: abort at /home/david/esp/esp-idf/components/esp32/./panic.c:572
0x400d0b4f: __cxa_end_catch at /home/david/esp/esp-idf/components/cxx/./cxx_exception_stubs.cpp:13
0x4012f998: operator new(unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_op.cc:54
0x4012f95d: operator new[](unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_opv.cc:32
0x40125710: nvs::PageManager::load(unsigned int, unsigned int) at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_pagemanager.cpp:24 (discriminator 4)
0x401244b6: nvs::Storage::init(unsigned int, unsigned int) at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_storage.cpp:41
0x40123d23: nvs_flash_init_custom at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_api.cpp:454
0x40123d8d: nvs_flash_init_partition at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_api.cpp:454

Is there any way to create an NVS on a large flash partition? To be clear, I do not want to store large chunks of data (each one is less than 200 bytes) and I like NVS very much for its efficiency and simplicity.

If there is no way, then I probably need create a SPIFFS partition...

Thank you for your help, you guys rock!

[chegewara]

I dont remember exact numbers, maybe you will find it somewhere on this forum, but nvs partition is using some heap. The bigger nvs partition is the more heap is reserved.

[Ritesh]

Hi,

our IoT product usually sends a steady stream of data to our server. If we lose connectivity to the server, we buffer the messages in NVS. To be able to bridge longer offline periods without any data loss, we want to use a large partition for NVS, which is why we upgraded our flash memory to 16 MB.

Unfortunately, I noticed that when I increase the NVS partition size, the chip won't start any more due to an out-of-memory error. Looking into the nvs_pagemanager.cpp file, it seams that the NVS system tries to allocate RAM for each page, which exhausts the heap. It works for a 1MB NVS partition, but fails for 2MB or larger partitions.

The backtrace (for a 6MB NVS partition):

Code: Select all

0x400d0b4f: __cxa_end_catch at /home/david/esp/esp-idf/components/cxx/./cxx_exception_stubs.cpp:13
0x40089273: invoke_abort at /home/david/esp/esp-idf/components/esp32/./panic.c:572
0x4008929f: abort at /home/david/esp/esp-idf/components/esp32/./panic.c:572
0x400d0b4f: __cxa_end_catch at /home/david/esp/esp-idf/components/cxx/./cxx_exception_stubs.cpp:13
0x4012f998: operator new(unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_op.cc:54
0x4012f95d: operator new[](unsigned int) at /builds/idf/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/new_opv.cc:32
0x40125710: nvs::PageManager::load(unsigned int, unsigned int) at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_pagemanager.cpp:24 (discriminator 4)
0x401244b6: nvs::Storage::init(unsigned int, unsigned int) at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_storage.cpp:41
0x40123d23: nvs_flash_init_custom at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_api.cpp:454
0x40123d8d: nvs_flash_init_partition at /home/david/esp/esp-idf/components/nvs_flash/src/nvs_api.cpp:454

Is there any way to create an NVS on a large flash partition? To be clear, I do not want to store large chunks of data (each one is less than 200 bytes) and I like NVS very much for its efficiency and simplicity.

If there is no way, then I probably need create a SPIFFS partition...

Thank you for your help, you guys rock!

Hi,

I think it's better to use SPIFFS file system for high data amount storage purpose instead of using NVS.

[cashchew]

Large nvs partition does indeed use a lot of heap. Is there a way to force the heap allocation to use external SPI RAM? SPIFFS is very slow when it comes to read speed if the partition is >2M, and getting a file written successfully is becoming less and less reliable when the capacity usage is 50% and above. Fatfs is easily get corrupted with power outage while a file is open for write, and there are times failed to be remount on next reset.

[Ritesh]

Large nvs partition does indeed use a lot of heap. Is there a way to force the heap allocation to use external SPI RAM? SPIFFS is very slow when it comes to read speed if the partition is >2M, and getting a file written successfully is becoming less and less reliable when the capacity usage is 50% and above. Fatfs is easily get corrupted with power outage while a file is open for write, and there are times failed to be remount on next reset.

Hello,

You can modify linker script to allocate heap memory using PSRAM.