ESP32 Forum

Posted: **Wed Nov 27, 2024 2:34 am**

As I currently understand it, Rainmaker devices transmit MQTT messages to AWS on a single topic, using JSON in the payload to distinguish the particular updated parameters.

If I'm right, then it leads to a >10x increase in the data usage for the device, compared with using the MQTT topic to distinguish the parameter.

This is not a problem for wifi connected devices, nor for the application which likely has a data plan on a mobile phone.

BUT this is a BIG problem for metered device applications, eg for Rainmaker devices in the field like agriculture or parcel tracking, which have to pay per byte on a Cellular IoT network.

@ESP_Piyush can you help me out here, hopefully by clearing up my misunderstanding, because Rainmaker looks like a really well thought out and secure system which is otherwise perfect for our needs.

Posted: **Wed Nov 27, 2024 1:53 pm**

Including param name in topic name and just the value in payload would be suitable for cases wherein the solution is just an MQTT publish/subscribe model.

However, ESP RainMaker does much more than that. There is a large node config structure which tells the phone apps what all parameters are supported by the node, what are their data types, permissions, etc. The node params structure being a JSON allows us to report multiple parameters in a single message and it being JSON allows us to use JSON patching, wherein you just report the changed params and the cloud backend merges them with the rest. The same are also convenient for consuming by phone apps and Alexa/Google integrations. Moreover, most of the decisions are also linked to the AWS pricing itself, which charges MQTT messages in increments of 5KB, so splitting param update info between topic name and payload have no real benefits in RainMaker's context.

Posted: **Thu Dec 05, 2024 1:20 pm**

Thanks for the detailed reply Piyush.

I think there are two separate issues here relating to reducing the WAN traffic for device metric updates

1. Whether it is technically feasible
2. Whether there is a valid business case

The business case is clear because for IoT in the field, eg agricultural monitoring, vehicle tracking etc the only backhaul available is cellular which is charged by MB, and a 10-20x increase in usage using raw JSON is not viable.

On the technical feasibility, I dont see why the AWS Lambda that processes the incoming data from the devices cannot translate a binary encoded metric back to its JSON representation before storage or further processing for the HTTP API

Ie if you use TLV, like Matter does, and store the schema in the AWS database, you can reconstruct the JSON on the fly using that schema eg

Code: Select all

 
import struct
import json

# Example schema (retrieved from the database)
device_schema = {
    "0x00": { "name": "device_id", "type": "string" },
    "0x01": { "name": "temperature", "type": "int16", "scaling_factor": 0.01, "unit": "C" },
    "0x02": { "name": "humidity", "type": "int16", "scaling_factor": 0.01, "unit": "%" },
    "0xFF": { "name": "timestamp", "type": "uint64", "unit": "epoch" }
}

def parse_tlv_dynamic(data, schema):
    parsed = {}
    i = 0
    while i < len(data):
        tag = f"0x{data[i]:02X}"  # Convert tag to hex
        length = data[i + 1]
        value = data[i + 2:i + 2 + length]
        
        if tag in schema:
            field = schema[tag]
            if field["type"] == "string":
                parsed[field["name"]] = value.decode()
            elif field["type"] == "int16":
                parsed[field["name"]] = struct.unpack(">h", value)[0] * field.get("scaling_factor", 1)
            elif field["type"] == "uint64":
                parsed[field["name"]] = struct.unpack(">Q", value)[0]
        i += 2 + length
    return parsed

# Example TLV payload
tlv_payload = b"\x00\x09sensor123\x01\x02\x08\x66\x02\x02\x11\x94\xFF\x08\x64\xA5\x90\x5F\x00\x00\x00"
parsed_data = parse_tlv_dynamic(tlv_payload, device_schema)
print(json.dumps(parsed_data, indent=2))

Posted: **Thu Dec 05, 2024 2:21 pm**

This logic would require the cloud to understand the schema and do additional processing, whereas like I mentioned, the current payloads are directly consumed by the phone apps via the REST APIs whose format is also JSON, which is convenient for consumption by the different clients. Moreover, use of JSON was also in-line with AWS IoT Shadows concept, wherein nodes directly write to the shadows, without going through any custom backend logic

As of now, one option is to use shorter device/param names so that the param updates, which is the most frequent message is shorter in length. Eg. {"D":{"T":34.5}} for a float value of 34.5 being reported for parameter T which is inside a device D.

We have TLV payloads for command-response but even there, the focus is not for reducing message size, but to enable binary data exchange. The request id, command id and other fields that are appended will add to the size of the raw binary payload.

We do not have any plans to support data formats other than JSON for param updates workflow.

Posted: **Sun Dec 08, 2024 11:32 am**

Thanks very much Piyush, will try your suggestion

ESP32 Forum

Problem using Rainmaker on metered connections

Problem using Rainmaker on metered connections

Re: Problem using Rainmaker on metered connections

Re: Problem using Rainmaker on metered connections

Re: Problem using Rainmaker on metered connections

Re: Problem using Rainmaker on metered connections