How to transmit video faster via wifi?

[roctwo]

Hi!
Here is my part of code:
static void udp_thread(void *p)
{
int i = 0;
int sock;
struct sockaddr_in toAddr;
xEventGroupWaitBits(wifi_event_group, CONNECTED_BIT, false, true, portMAX_DELAY);
LWIP_UNUSED_ARG(p);
sock = socket(AF_INET,SOCK_DGRAM,IPPROTO_UDP);
if(sock < 0)
{
ESP_LOGI(TAG, "socket err");
}
memset(&toAddr,0,sizeof(toAddr));
toAddr.sin_family=AF_INET;
toAddr.sin_addr.s_addr=inet_addr("192.168.10.9");
toAddr.sin_port = htons(REMOTE_PORT);
while(1)
{
sendto(sock,start,SEND_START_LEN,0,(struct sockaddr*)&toAddr,sizeof(toAddr));
vTaskDelay(10/portTICK_RATE_MS);
for(i=0; i<240; i++)
{
sendto(sock,image1+i*480,SEND_BUF_LEN,0,(struct sockaddr*)&toAddr,sizeof(toAddr));
vTaskDelay(10/portTICK_RATE_MS);
}

sendto(sock,end,SEND_END_LEN,0,(struct sockaddr*)&toAddr,sizeof(toAddr));
vTaskDelay(10/ portTICK_RATE_MS);
}
close(sock);
vTaskDelete(NULL);
}
-----------------------------------------------------------------------------------------------------------------

Every time after I send a udp packet I have to call vTaskDelay.It is because that wifi task need CPU to handle and idle task need to feed watch dog,so my task can't always occupy CPU.And if I shorten the delayed time,some udp packets will lost.So the time spent in vTaskDelay is a big waste，I can't find a way to avoid this.
My question is :
Are there some methods to transmit video as far as ESP32 can?

Thanks!

[kolban]

I haven't researched this but maybe the use of the select() sockets call might be what you need. For any given socket, we can ask:

1) Does the socket have some data to read()?
2) Is the socket ready to be written?
3) Is there an exception on the socket?

The select system call can be configured to "block" until such time as any of these are true.

Looking at option (2) above, this might be what you need. If you call select() asking if the socket you are using is ready to be written, then it should "block" until it is ready to be written. So now instead of waiting a time period, you will be waiting until such time as the socket is ready to accept more data for transmission. I also believe that select() is "FreeRTOS" aware meaning that if we block in select(), it will happily allow context switches to other FreeRTOS tasks without "busy consuming" the CPU.

Again ... just a guess ...

[ESP_Sprite]

Are you sure you need to call vTaskDelay? The sending of packets happens asynchroneously, that is, while the write is busy, the WiFi hardware probably is working on sending the packets and your thread will get descheduled automatically, making room for the idle task to be called.

[roctwo]

I need a API which can register the callback function of wifi's sending success.Does this API exists? If not,do you plan to provide this API?

[ESP_Sprite]

There is no such API: normally a send is blocking, that is, the function will not return until the packet is queued for sending.

Edit: Looking at your original code, I don't see why removing the vTaskDelay()s should lead to trouble. Just out of curiosity, can you see what happens when you remove them and start the thread on core 1 by using vTaskCreatePinnedToCore?

[WiFive]

There is no such API: normally a send is blocking, that is, the function will not return until the packet is queued for sending.

Edit: Looking at your original code, I don't see why removing the vTaskDelay()s should lead to trouble. Just out of curiosity, can you see what happens when you remove them and start the thread on core 1 by using vTaskCreatePinnedToCore?

The first two, anyway. I don't see anything that waits for a new frame of video to be ready, so maybe should simulate 10fps with a 100ms delay.

8bits/pixel * 240 * 480 * 10fps = 9.2Mbps

[roctwo]

If I remove the vTaskDelay, many UDP packets will be lost.

[kolban]

Can you elaborate on your design? Are you sending video data using a known format or protocol? Does it have to be datagram or can it be stream (TCP)? What is the source of the video on the ESP32? What is the nature of the application/system receiving the data stream? Is there any video compression being used? Can the system tolerate "lost" fragments of data?

[roctwo]

Can you elaborate on your design? Are you sending video data using a known format or protocol? Does it have to be datagram or can it be stream (TCP)? What is the source of the video on the ESP32? What is the nature of the application/system receiving the data stream? Is there any video compression being used? Can the system tolerate "lost" fragments of data?

OK,My project has changed a lot in the above code.But I met again about UDP's sending packet problem.The demand of my project is to collect the data of camera by ESP32, and then send it to PC in real time by wifi. LAN transmission.It is similar to demand for real-time monitoring.
My design is :ESP32 collects camera data through I2S protocol.I set the I2S DMA ping-pang buffer.When a buffer is full, I2S interrupt will occuer.In the ISR, let the camera data fill into another buffer, and then notify the UDP thread to send the data from the DMA buffer which has just been filled to PC through wifi.
Here is my i2s_isr code:

Code: Select all

static void IRAM_ATTR i2s_isr(void* arg)
{
	I2S0.conf.rx_start = 0;
	I2S0.int_clr.val = I2S0.int_raw.val;       
	/******************switch buffer to let camera data fill***************************/              
    	s_cur_buffer = !s_cur_buffer;
  	if (s_isr_count == 239) 
	{
		i2s_stop();
    	}
	else 
    	{
    		++s_isr_count;
	}
	i2s_fill_buf(s_cur_buffer);
	BaseType_t xHigherPriorityTaskWoken;
	/******************notify udp_thread***************************/     
    	xSemaphoreGiveFromISR(s_data_ready, &xHigherPriorityTaskWoken);
    	if (xHigherPriorityTaskWoken != pdFALSE) 
	{      
        	portYIELD_FROM_ISR();
    	}
}

Here is my UDP_thread code:

Code: Select all

void udp_thread(void *p)
{
		int prev_buf = -1;
		int sock;
		struct sockaddr_in toAddr;
		LWIP_UNUSED_ARG(p);
		sock = socket(AF_INET,SOCK_DGRAM,IPPROTO_UDP);
		if(sock < 0)
		{
			ESP_LOGI(TAG, "socket err");
		}
		memset(&toAddr,0,sizeof(toAddr));
		toAddr.sin_family=AF_INET;
		toAddr.sin_addr.s_addr=inet_addr("192.168.23.1");
		toAddr.sin_port = htons(REMOTE_PORT);
		while (true) 
		{
                        /******************waiting notification from udp_i2s_isr***************************/     
			xSemaphoreTake(s_data_ready, portMAX_DELAY);
                        FD_ZERO(&rdfds); 
			FD_SET(sock,&rdfds); 
			usertimeout.tv_sec=0;
			usertimeout.tv_usec=1000;
                        /******************buf_idx is the buffer which has just been filled fully ***************************/   
			int buf_idx = !s_cur_buffer;
			if (prev_buf != -1 && prev_buf == buf_idx) 
			{	  
				ets_printf("! %d\n", s_line_count);
			}
                        const uint32_t* buf = s_dma_buf[buf_idx];
			prev_buf = buf_idx;
                        /******************A frame has been collected fully from  camera***************************/  
			if (!s_i2s_running) 
			{
				prev_buf = -1;
				s_frame_count++;
                                /******************frame_end tells the receiver that this is the end of this frame***************************/   
				sendto(sock,frame_end,SEND_END_LEN ,0,(struct sockaddr*)&toAddr,sizeof(toAddr));
                                /******************new fram collect***************************/ 
				i2s_run(s_fb_w, s_fb_h);
                               /******************frame_start tells the receiver that this is the start of this frame***************************/   
				sendto(sock,frame_start,SEND_START_LEN ,0,(struct sockaddr*)&toAddr,sizeof(toAddr));
				continue;
			}
			ret=select(sock+1,NULL,&rdfds,NULL,&usertimeout);

			if(ret < 0) 
			{
				perror("select error\n");
			}
			else if(ret == 0)
			{
				printf("time out\n"); 
			}
			else
			{ 
		        	if(FD_ISSET(sock,&rdfds))   
				{
				/******************send the part of centent of frame.***************************/  
					sendto(sock,buf,960,0,(struct sockaddr*)&toAddr,sizeof(toAddr));
                                        ++s_line_count;
				}
        		
    		         }
	      }
				close(sock);
				vTaskDelete(NULL);
}

My problem is:According to my debugging,every sendto API that should be called has been called.But From my observation on Wireshark,more than half of the UDP package is missing.The time that sento API needs to run is more than sufficient.
I don't know the detail about wifi stack in ESP32,So I hope that engineers from espressif can share their views.It will helps our team a lot.
Can you share your opinion about it.I will appreciate it very much.

[ESP_igrr]

Do you have some statistics, how much data and how often is the application sending to the UDP stack?
Which core is your line filter task running on? Is FreeRTOS running in single core or dual core mode?