A Step-by-Step guide getting mutual authentication and communiation with ESP32-AT FW over REST to work with Amazon's AWS

[flosko]

Hi there,

having again invested a couple of hours to sort out the pitfalls of getting my ESP32-AT FW-based things (FW version V1.1.3) to work with Amazon's IoT platform, I' d like to share my insights. I guess it might save the one or the other from similar troubles I had, mainly caused by *not* using one of the available SDKs.

In a nutshell, you need to make sure that you:

1. Create thing: Create the device certificate and private key on AWS, attach a suitable policy to allow access, don'T forget to activate the certificate.
2. Apply certs: Convert the device certificate, key and ca cert to the binary format and flash the produced files into the ESP32 module .
3. Time settings: Fetch time from SNTP, otherwise authentication won't work (certificate life time in the future)
4. Prepare connection: For ESP32-AT FW use "AT+CIPSSLCCONF=3,0,0" to enable mutual authentication.
5. Establish connection: Use Port 8443 (*not* 443) to communicate with AWS over REST.
6. Communicate: Use GET to get device shadow, use POST to update shadow.

LONG VERSION:


1. Create thing:
On AWS create an object (my object, aka "thing" is called "lnu_001" in the examples below), furthermore create a certificate for the device and a key pair. ASSOCIATE the certificate with the object. Download the device's certificate (a .crt-file), the device's private key (a .pem.key file), and one of the Amazon root certificates (a .pem-file to authenticate the server towards the client). Furthermore create a device policy, which might look in the simplest case as the one below (simply allow everything) and ATTACH it to the certificate. It's further paramount to ACTIVATE the certificate.

Code: Select all

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:*",
      "Resource": "*"
    }
  ]
}

Once you have created an object/thing and associated it with both, a certificate and a policy you are done. It's all nicely described here: https://docs.aws.amazon.com/iot/latest/ ... ot-gs.html


2. Apply certs to ESP32-AT:
You downloaded a client certificate, a private key and a RootCA certificate just before, which need to be put into the ESP32. You can use the Flash_download_tool from espressif for that: https://www.espressif.com/en/support/do ... ther-tools
However, the certificates are in a text format and not in the required binary format, so we need to convert them. You can achieve this with the Python script AtPKI.py in the tools-folder of the ESP32-AT FW. Be careful, there are different type parameters depending on if you convert a certificate, key or CA.

Code: Select all

// follow instructions from https://github.com/espressif/esp32-at/blob/master/tools/README.md
// convert device certificate to bin format
$ python ../../esp32-at/tools/AtPKI.py generate_bin -b client-cert.bin cert e47a1ec726-certificate.pem.crt.txt

// convert private key to bin format
$ python ../../esp32-at/tools/AtPKI.py generate_bin -b private_key.bin key e47a1ec726-private.pem.key

// convert Amazon root CA to bin format
$ python ../../esp32-at/tools/AtPKI.py generate_bin -b amazon-ca.bin ca AmazonRootCA1.pem

Once converted, you take the binary files and flash them with the programming tool into *specific* locations:

Code: Select all

# Name	        Type 	SubType  	Offset		Size
client-cert	0x40	   	5	0x2a000		8K
private_key	0x40	   	6	0x2c000		8K
amazon-ca	0x40	   	7	0x2e000		8K

3. Time settings:
Now we are good to go and test our setup. I assume your ESP32-AT is connected to a WiFi and has Internet connectivity.

The time on our board running ESP32-AT FW must be set correctly. Otherwise, our TLS client can't validate the server certificate, because the current date/time is outside the validity period of the certificate.

So, we make our module fetch the time from an NTP server using SNTP and check whether its current or not

Code: Select all

// enable SNTP, configure time zone (here 1 hour to the East), and specify an NTP server
AT+CIPSNTPCFG=1,1,"us.pool.ntp.org"#013#010

// give it some seconds and check if we got a current time
AT+CIPSNTPTIME?#013#010

4. Prepare connection:
Configure your device for mutual authentication, i.e., our module presents its certificate to the server which checks if it is allowed to connect *and* the server issues a certificate to our device which validates it against the known CA.

Code: Select all

//set mutual authentication; assume there is only one client cert and one CA, so both have the index 0
AT+CIPSSLCCONF=3,0,0#013#010

5. Establish connection:
It's paramount to connect to port 8443 and *not* 443, when using RESTful services. The tricky part here is, you even can connect to 443 but will receive repeated HTTP 403 error codes ("forbidden") and you might start your search for the problem at the wrong place (certificate, policies, keys...).

So always use 8443 and you are good.

Code: Select all

// establish TLS connection to Amazon AWS
// ATTENTION: for RESTful services use Port 8443
AT+CIPSTART="SSL","xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com",8443
cert_len=1224
prvtkey_len=1679
ca_len=1188
ctx ok

CONNECT

OK

Code: Select all

//if the time is not set correctly, you may experience this behavior:
AT+CIPSTART="SSL","xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com",8443
cert_len=1224
prvtkey_len=1679
ca_len=1188
ctx ok
ssl connect fail

ERROR

6. Communicate:
Once you are successfully connected you can use simple POST- and GET-requests to communicate with the created thing (in our case "lnu_001").

Let's start with an update request. Use POST to push a new value of attribute "prop1" to device "lnu_001"

Code: Select all

// Update thing "lnu_001" property "prop1" to value "12"
AT+CIPSEND=261

OK

> POST /things/lnu_001/shadow HTTP/1.1
Host: xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com
Content-Type: application/json  
Content-Length: 109

{
    "state": {
        "desired": { "prop1":"12"
        },
        "reported": {
        }
    }
}
Recv 261 bytes
id:0,Len:261,dp:0x3ffc3b98

SEND OK

+IPD,189:HTTP/1.1 200 OK
content-type: application/json
content-length: 157
date: Fri, 22 Feb 2019 13:45:11 GMT
x-amzn-RequestId: 09d6baa8-7cd9-24ff-cbef-654cbf95a59d
connection: keep-alive


+IPD,157:{"state":{"desired":{"prop1":"12"},"reported":{}},"metadata":{"desired":{"prop1":{"timestamp":1550843111}},"reported":{}},"version":7,"timestamp":1550843111}

Now as we have set "prop1" to value "12" we would like to retrieve from AWS IoT if this value has been indeed stored in the cloud. Therefore, we retrieve *all* properties of "lnu_001" (which has besides "prop1" a further one called "temp").

Code: Select all

// Get all properties of thing "lnu_001", including property "prop1"
AT+CIPSEND=93

OK

> GET /things/lnu_001/shadow HTTP/1.1
Host: xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com
Recv 93 bytes
id:0,Len:93,dp:0x3ffc3b98

SEND OK

+IPD,189:HTTP/1.1 200 OK
content-type: application/json
content-length: 208
date: Fri, 22 Feb 2019 13:46:07 GMT
x-amzn-RequestId: d8b334f6-13e3-4964-e523-8039b55633c4
connection: keep-alive


+IPD,208:{"state":{"desired":{"temp":"32","prop1":"12"},"delta":{"temp":"32","prop1":"12"}},"metadata":{"desired":{"temp":{"timestamp":1550840430},"prop1":{"timestamp":1550843111}}},"version":7,"timestamp":1550843167}

That's it, folks. Amazon promised "connect your IoT device in 5 minutes". It took me more than a day, however, not using an SDK (which means you need to craft your HTTP requests manually) and fighting a bit with the AT command set of ESP32-AT FW may take a while).

cheers!

[tcpipchip]

Hi, how did you compute the content-lenght size and AT+CIPSEND parameters ?

And wich serial terminal software used ?

thanks