Server Receive and Send from 100 ESP32 Device

[Mahmoud Ahmed]

Hello everyone.

I'm new here and need help to start a big project.
I have built a web server on the ESP32 sensor and need to upgrade it .

My Question
What is the best way to connect 100 Device ( esp32 or esp8266 ) to the server , the Server and 100 ESP32 device connected to the wifi network with a router
The server will be installed on raspberry pi or windows 10 pc and store incoming data on SQLite database .
The server can send and receive from all ESP device on the network .
each ESP Device will send data to server every 1 Sec and it is not possible to exceed this period .
how can the server receving all data from each device without lose any thing .
i need only the guide step for search .
Thanks

[lbernstone]

https://mqtt.org/
https://iotbytes.wordpress.com/store-mq ... -database/
Use a usb hard disk. This level of activity will wear out an sdcard quickly.

[Mahmoud Ahmed]

Thanks, lbernstone

https://mqtt.org/
https://iotbytes.wordpress.com/store-mq ... -database/

I am planning to use this MQTT protocol but my question is can MQTT protocol handle over than 100 of ESP32 to communicate with one server and each of this send every 1 sec ? ?

Use a USB hard disk. This level of activity will wear out an SD card quickly.

Yes, this is the best solution

[mrburnette]

...
I am planning to use this MQTT protocol but my question is can MQTT protocol handle over than 100 of ESP32 to communicate with one server and each of this send every 1 sec ? ?
...
The server will be installed on raspberry pi or windows 10 pc and store incoming data on SQLite database .

It would be 'best' to ask over at the MQTT site:
https://groups.google.com/g/mqtt

as your desired answer is a "server-side" concern involving the Raspberry Pi host and the server. The client side has its own concerns regarding retransmission of TCP packets under the higher-level MQTT messaging format; that is, random transmission of 100 clients every second may be an issue depending on multiple client-side concerns around re-transmit (especially if on same WiFi LAN.)

As a "few" test clients will not scale linearly, I would approach this project with some suspicions on a clean and successful implementation.

[lbernstone]

Short answer, yes, a pi3 or better can handle 100 entries per second.
Long answer, there are many ways to increase capacity of a service like mqtt. haproxy can easily be configured to distribute the load between a couple of mqtt servers, with the added benefit of high availability in case one device fails. Google knows lots about load balancing and high availability.