TAMP compression technique for JSON packets.

[fanmen1]

Update on TAMP compression, the compression works fine. However, the decompression bytes are way out of line. For example:
Compression successful!
Original size: 9890 bytes
Compressed size: 2005 bytes
will return the decompression of:
Decompression successful!
Decompressed data:
Decompressed size: 1402438 bytes
I've tried to adjust the buffer size and the window size as needed but this is always the end result. Any suggestions on this?
Below is the function I'm using for the decompression, and I'm using the tamp_compressor_compress_and_flush to compress & flush out all the remainig data. tamp_res decompress_json_packet_cb(const unsigned char *compressed_data, size_t compressed_size, unsigned char **decompressed_data, size_t *decompressed_size) {
TampDecompressor decompressor;
TampConf conf = {.window=10, .literal=8, .use_custom_dictionary=0}; // Example configuration; adjust as needed
unsigned char window[WINDOW_SIZE];
unsigned char output[BUFFER_SIZE];
size_t output_written_size = 0;
size_t input_consumed_size = 0;

Buffer buffer = {NULL, 0, 0};

// Initialize the decompressor
tamp_res res = tamp_decompressor_init(&decompressor, &conf, window);
if (res != TAMP_OK) {
return res;
}

size_t remaining_input_size = compressed_size;
const unsigned char *current_input = compressed_data;

while (remaining_input_size > 0) {
size_t chunk_size = remaining_input_size > BUFFER_SIZE ? BUFFER_SIZE : remaining_input_size;

res = tamp_decompressor_decompress_cb(
&decompressor,
output,
BUFFER_SIZE,
&output_written_size,
current_input,
chunk_size,
&input_consumed_size,
append_to_buffer,
&buffer
);

if (res != TAMP_OK && res != TAMP_OUTPUT_FULL && res != TAMP_INPUT_EXHAUSTED) {
if (buffer.data) {
heap_caps_free(buffer.data);
}
return res;
}

remaining_input_size -= input_consumed_size;
current_input += input_consumed_size;
}

*decompressed_data = buffer.data;
*decompressed_size = buffer.size;
return TAMP_OK;
}

[fanmen1]

Documentation for TAMP compression technique: [url][/https://tamp.readthedocs.io/en/latest/c_library.html]

[MicroController]

Code: Select all

unsigned char window[WINDOW_SIZE];
unsigned char output[BUFFER_SIZE];

may be too big to be placed in local variables, i.e. on the stack.

Notice that the callback is only called for status reporting during decompression ("% of input already processed"). You'll have to use output_written_size to determine the size of the current decompressed chunk, and the decompressed data is put into output.

[fanmen1]

I realised that, hence moved everything to PSRAM as below:
unsigned char *window = (unsigned char *)heap_caps_malloc(WINDOW_SIZE, MALLOC_CAP_SPIRAM);
unsigned char *output = (unsigned char *)heap_caps_malloc(BUFFER_SIZE, MALLOC_CAP_SPIRAM);

[fanmen1]

If anyone looking for decompression algorithm for their JSON packets, give a brotli compression a try: [url][/https://github.com/martinberlin/brotli/ ... e/compress]. Very simple and effective.