[JavaScript] JSON parsing faster alternative

KanyeKanye
Posts: 54
Joined: Mon Dec 05, 2016 12:34 am

[JavaScript] JSON parsing faster alternative

Postby KanyeKanye » Wed Sep 25, 2019 11:58 am

My esp32 chip is used for recording some important data into local memory. I need to send these data to the user webbrowser for drawing a chart. It is about 1MB per hour and I need charts with up to 5days of data (up to 120MB of raw data).

At the moment I have written an http server and JSON parser. When user requests data, raw data from memory are read and parsed into JSON for sanding. Parsing raw data into text json result in extending each 1MB (of raw) into about 2MB (of text).

Local browser waits for JSON file and using JS JSON.parse(this.responseText) parses it into object for adding into an javascript array.
JSON.parse tooks lots of time! About 5 sec for each 2MB (1MB of raw).

Code: Select all

function loadChartData(address) {
    var xhttp = new XMLHttpRequest();
    xhttp.onreadystatechange = loadChartDataReady;
    xhttp.open("GET", "http://192.168.1.1/"+address, true);
    xhttp.send();
}
function loadChartDataReady() {
    if (this.readyState != 4) return;
    if (this.status != 200) { return; }
    try {
        var json = JSON.parse(this.responseText);
        for(var i = 1; i < json.length; i++) {
            var obj = json[i];
            series.splice(i - 1, 0, {ts: obj[0], time: obj[1], s: obj[2], m: obj[3], on: obj[4]} );
        }
        updateChart();
    } catch(err) {
        console.log("error: " + err);
    }
}
I am wandering what can I do for speeding up my code both in terms of used memory and execution time.
  • Can I parse raw data somehow in javascript?
  • Can I change JSON.parse into some another function for speeding up?
  • Can I change XMLHttpRequest into something another for start parsing data while they are still being sent?

tommeyers
Posts: 184
Joined: Tue Apr 17, 2018 1:51 pm
Location: Santiago, Dominican Republic

Re: [JavaScript] JSON parsing faster alternative

Postby tommeyers » Wed Sep 25, 2019 2:01 pm

You can send a body in the html that is not json.

See: https://developer.mozilla.org/en-US/doc ... P/Messages

If you do that you will reduce the load on the esp32 (memory and processors), shorten the transmit time; on the browser you will simplify and reduce the cpu load for json processing.

I think you could also split your body across multiple html messages. That would make concurrent (server browser) processing possible. I can't find an example of that with a quick look.

Tom
IT Professional, Maker
Santiago, Dominican Republic

PeterR
Posts: 621
Joined: Mon Jun 04, 2018 2:47 pm

Re: [JavaScript] JSON parsing faster alternative

Postby PeterR » Wed Sep 25, 2019 7:16 pm

You are using Ajax (Ajaj...).
1) You could chunk requests, for example:

Code: Select all

GET /api/v1/data?start=x&end=y
You may then extract the HTTP query start/end parameters using the ESP library & your webserver would simply serialise between ranges.

2) Custom format
As suggested, you do not need to encode as JSON. You could send CSV. Maybe the first line is a header and following lines are the data.
If you split the header line using the comma deliminator into array headerNames then you will have the Javascript property names.
So:

Code: Select all

myObj[headerNames[i]] = value
I doubt that you will gain in total time but as you are looping on an array of lines you may update your chart whenever you like.

3) Raw data
Be careful. Encode data into network endian . Text has certain advantages!

3) Websocket
A Websocket would allow you to send data in smaller chunks and in real-time.
The ESP webserver does not support websockets but the Asynchronous webserver does.

I would be tempted to use Websockets but a combination of 1) and 2) would be simplest to get running.
& I also believe that IDF CAN should be fixed.

KanyeKanye
Posts: 54
Joined: Mon Dec 05, 2016 12:34 am

Re: [JavaScript] JSON parsing faster alternative

Postby KanyeKanye » Thu Sep 26, 2019 5:31 pm

I've decided to send raw data. Bytes are now read and send to the browser without any parsing. That way I could use larger buffers and read more from memory to send. Whats more I know exactly how much data will be send.
On browser, javascript side I parse data using DataView, getUint32, getInt8

tommeyers
Posts: 184
Joined: Tue Apr 17, 2018 1:51 pm
Location: Santiago, Dominican Republic

Re: [JavaScript] JSON parsing faster alternative

Postby tommeyers » Thu Sep 26, 2019 10:46 pm

So, you intend to send data that potentially has byte values 0..255?

If you do you will be back here.

Tom
IT Professional, Maker
Santiago, Dominican Republic

PaulNi
Posts: 41
Joined: Tue Nov 07, 2017 3:50 pm

Re: [JavaScript] JSON parsing faster alternative

Postby PaulNi » Fri Sep 27, 2019 6:54 am

I think getting it parsed faster will not help much.

You biggest concern should be the IO bottleneck. Think of streaming the json to some parser before rewriting the parser itself

PeterR
Posts: 621
Joined: Mon Jun 04, 2018 2:47 pm

Re: [JavaScript] JSON parsing faster alternative

Postby PeterR » Fri Sep 27, 2019 11:17 am

If you use RAW bytes then just make sure you encode/decode using the correct endians!
& I also believe that IDF CAN should be fixed.

Who is online

Users browsing this forum: No registered users and 59 guests