Using Miniz compression for JSON packet.

[fanmen1]

Hi, I'm trying to link the miniz compression algorithm according to the example shown here https://github.com/richgel999/miniz/blob/master/miniz.c

I have no problem compiling the code, but the linker is throwing the error shown below. The function its pointing to as the 'undefined reference' is defined correctly under miniz_tdef.h header file. Any suggestions on this issue?

[Bryght-Richard]

> The function its pointing to as the 'undefined reference' is defined correctly under miniz_tdef.h header file.

I think

Code: Select all

tdefl_create_comp_flags_from_zip_params

is only declared in miniz_tdef.h, but is likely defined elsewhere, likely in a '.C' file. Have you added the corresponding C files, like miniz_tdef.c, to your project?

[Horvat]

You've probably forgot to include all source files into your CMakeLists.txt, that is miniz.c miniz_tdef.c miniz_tinfl.c miniz_zip.c
After a short while, I was able to successfully build example1

[MicroController]

Core routines of miniz are present in ROM (see viewtopic.php?t=1076) and should actually be linked from there.

[MicroController]

Btw, you may want to also try a less RAM demanding library than miniz, see e.g. viewtopic.php?f=2&t=39798

[fanmen1]

Thanks guys, you're right it was stupid mistake.
As you mentioned, yes miniz is very RAM hungry method for a mediocre compression.
I'm now making use of the Tamp, which is definitely better than miniz, so thanks for the info.

[fanmen1]

Update on TAMP compression, the compression works fine. However, the decompression bytes are way out of line. For example:
Compression successful!
Original size: 9890 bytes
Compressed size: 2005 bytes
will return the decompression of:
Decompression successful!
Decompressed data:
Decompressed size: 1402438 bytes
I've tried to adjust the buffer size and the window size as needed but this is always the end result. Any suggestions on this?
Below is the function I'm using for the decompression, and I'm using the tamp_compressor_compress_and_flush to compress & flush out all the remainig data. tamp_res decompress_json_packet_cb(const unsigned char *compressed_data, size_t compressed_size, unsigned char **decompressed_data, size_t *decompressed_size) {
TampDecompressor decompressor;
TampConf conf = {.window=10, .literal=8, .use_custom_dictionary=0}; // Example configuration; adjust as needed
unsigned char window[WINDOW_SIZE];
unsigned char output[BUFFER_SIZE];
size_t output_written_size = 0;
size_t input_consumed_size = 0;

Buffer buffer = {NULL, 0, 0};

// Initialize the decompressor
tamp_res res = tamp_decompressor_init(&decompressor, &conf, window);
if (res != TAMP_OK) {
return res;
}

size_t remaining_input_size = compressed_size;
const unsigned char *current_input = compressed_data;

while (remaining_input_size > 0) {
size_t chunk_size = remaining_input_size > BUFFER_SIZE ? BUFFER_SIZE : remaining_input_size;

res = tamp_decompressor_decompress_cb(
&decompressor,
output,
BUFFER_SIZE,
&output_written_size,
current_input,
chunk_size,
&input_consumed_size,
append_to_buffer,
&buffer
);

if (res != TAMP_OK && res != TAMP_OUTPUT_FULL && res != TAMP_INPUT_EXHAUSTED) {
if (buffer.data) {
heap_caps_free(buffer.data);
}
return res;
}

remaining_input_size -= input_consumed_size;
current_input += input_consumed_size;
}

*decompressed_data = buffer.data;
*decompressed_size = buffer.size;
return TAMP_OK;
}

[fanmen1]

If anyone looking for decompression algorithm for their JSON packets, give a brotli compression a try: [url][/https://github.com/martinberlin/brotli/ ... e/compress]. Very simple and effective.