IP suddenly stops responding after several hours running

[mbratch]

I have two separate ESP32 Pico kits running a simple REST API server. The applications are identical in function. One is written with the Arduino framework. The other I ported to ESP-IDF.

I am using 'curl' on a Linux server in a script to generate REST API calls to each of the ESP32's every second just as an endurance test. I let these tests run simultaneously against each ESP32. I am sending requests to one ESP32 every second and the other every two seconds (half the rate).

The ESP32s respond fine to the REST API calls for a several hours (8 to 18) and then suddenly and simultaneously stop responding to any requests. Neither do they respond to 'ping'. I reset the ESP32s and they will start responding again with no changes to the client side (Linux test tool).

I finally did capture an error that appears on the console when the error occurs (IP stops responding):

Code: Select all

I (10605501) wifi:state: run -> init (ec0)
I (10605501) wifi:pm stop, total sleep time: 9582020957 us / 10604598223 us

I (10605501) wifi:new:<11,0>, old:<11,0>, ap:<255,255>, sta:<11,0>, prof:1
W (10605511) httpd_txrx: httpd_sock_err: error in recv : 113

It would seem to me that there must be something happening on the network that both ESP32s are reacting to that is for some reason killing the IP link. I can start up both ESP32s at totally different times, but they fail simultaneously. Be it known that this network has both Windows and Linux machines on it.

Has anyone encountered this before? I've done a lot of searching and have found one or two similar problems, but they were occurring within 10 or 15 minutes. In that case the issue has something to do with NetBIOS. Since I have Windows clients on my network, some mysterious network activities may be occurring.

[ESP_Sprite]

Possibly a memory leak? You may want to print the amount of free memory (using e.g. heap_caps_get_free_size(MALLOC_CAP_8BIT)) to see if that consistently goes down.

[mbratch]

Possibly a memory leak? You may want to print the amount of free memory (using e.g. heap_caps_get_free_size(MALLOC_CAP_8BIT)) to see if that consistently goes down.

Possibly. Although I'm sending packets to one ESP32 at double the rate that I'm sending them to the other, yet they stop responding at exactly (I mean EXACTLY) the same time. I've also had both of them run for nearly a full day before they stopped simultaneously. That's why I suspect there's something happening on the network that the ESP'32's are reacting to that I don't know about.

I'll try displaying the free memory during the test. That's a good suggestion. Ultimately, maybe WireShark...

[mbratch]

Possibly a memory leak? You may want to print the amount of free memory (using e.g. heap_caps_get_free_size(MALLOC_CAP_8BIT)) to see if that consistently goes down.

I ran `heap_caps_get_free_size(MALLOC_CAP_8BIT)` at regular intervals. The output was steady as a rock, so at least it did not show any sign of memory leak.

[mbratch]

I converted my application over to ESP-IDF and I still have the identical problem. Here's a message I see on the monitor when the wifi connection drops:

Code: Select all

I (10605501) wifi:state: run -> init (ec0)
I (10605501) wifi:pm stop, total sleep time: 9582020957 us / 10604598223 us

I (10605501) wifi:new:<11,0>, old:<11,0>, ap:<255,255>, sta:<11,0>, prof:1
W (10605511) httpd_txrx: httpd_sock_err: error in recv : 113

[MasterExploder]

Did you manage to solve this?

[mbratch]

Did you manage to solve this?

Yes.

Originally, I worked around the problem by writing my own wifi event handler in ESP-IDF and, when an unexpected disconnect occurred, I would reconnect to the AP.

As far as root cause... there's a problem that happens with only some AP routers. When I wrote my own handler, I displayed the reason code for the disconnect, which indicated a "MIC failure". I submitted a ticket on github, and all the details are on that ticket. I believe there was eventually a fix made in the ESP-IDF library. But I was unable to test whether it fixed the root cause in my case because I had since also changed my router, and my new router did not exhibit the issue.