ESP32 Forum

I'm writing a log file to an SD card. Right now, each entry has a fixed length (16 bytes). 1 data point is generated every second, and they're appended to the file en masse every 30 seconds. My issue is that if the device is turned off during the SD write (which is not visible to the user), the data may be corrupted partway through an entry.
The file is structured as an array (right now -- I could change it if need be): just each entry, in a row. Right now there is no data integrity checking, so there's no way to know if, say, one or two bytes extra got added right before a power off, and there's no way to know where the error is. That's what I would like to add now. I could just add a hash of each entry after it; the problem is, scanning through the entire file every time the device is turned on could take a long time.
Does anybody know how a good way to preserve data integrity in a file like this? Ensuring each write operation is atomic, efficiently checking hashes, etc. I tried Googling for a solution but could only find questions about specific systems, not about the data side.

There's nothing particularly system specific here.
The primary factor for data integrity is your filesystem. FAT is performance optimized, and has effectively no integrity checks. LittleFS has a good writeuphttps://github.com/littlefs-project/lit ... /DESIGN.md about this.
From a coding standpoint, align your writes to the sector size rather than an arbitrary 30 seconds. This minimizes the amount of writes (keeping the fs from doing rewrites), and stops fragmentation. Since your log entries are a fixed 16 bytes, this makes alignment/counting pretty easy, and gives you a simple sanity check- if the file size is not a multiple of the entry size, it is corrupt, and it should be the last entry that was corrupted.