memory management issue

[ves011]

I have an application built with esp-idf v 5.1.2 on ESP32-WROOM-32 where I redirect all messages (logs, printfs, …) over TCP, to a server running ncat.
So I have my own functions: my_log_vprintf(), my_printf(), my_fputs() which capture the output and send it to tcp, or to serial console, or disregard, depending on the value of console_state variable.

Code: Select all

int my_log_vprintf(const char *fmt, va_list arguments)
	{
	if(console_state == CONSOLE_ON)
		return vprintf(fmt, arguments);
	else if(console_state == CONSOLE_TCP)
		{
		char buf[1024];
		vsnprintf(buf, sizeof(buf) - 1, fmt, arguments);
		return tcp_log_message(buf);
		}
	return 0;
	}

void my_printf(char *format, ...)
	{
	char buf[1024];
	va_list args;
	va_start( args, format );
	vsnprintf( buf, sizeof(buf) - 1, format, args );
	va_end( args );
	if(console_state == CONSOLE_ON)
		{
		puts(buf);
		}
	else if(console_state == CONSOLE_TCP)
		{
		//puts(buf);
		tcp_log_message(buf);
		}
	}
void my_fputs(char *buf, FILE *f)
	{
	if(console_state == CONSOLE_ON)
		{
		puts(buf);
		}
	else if(console_state == CONSOLE_TCP)
		{
		//puts(buf);
		tcp_log_message(buf);
		}
	}

The issue comes when console_state == CONSOLE_TCP and the output is sent over tcp. The issue is about memory consumption which increase with every message sent over tcp and the available memory gets lower and lower and get into bigger problems
Here is my tcp_log.c which sends the buffer to the server running ncat

Code: Select all

static int tcp_log_enable;
static 	struct sockaddr_in srv_addr;
static TaskHandle_t tcp_log_task_handle;
static QueueHandle_t tcp_log_evt_queue;
static void tcp_log_task(void *pvParameters);
#define MSG_QUEUE_SIZE		20

int tcp_log_init(void)
	{
	int ret = ESP_FAIL;
	tcp_log_enable = 0;
	int sendsock;
	struct hostent *hent = NULL;
	if(!tcp_log_enable)
		{
		bzero((char *) &srv_addr, sizeof(srv_addr));
		sendsock = socket(AF_INET, SOCK_STREAM, 0);
		if(sendsock >= 0)
			{
			hent = gethostbyname(LOG_SERVER);
			if(hent)
				{
				srv_addr.sin_port = htons(LOG_PORT);
				srv_addr.sin_family = AF_INET;
				srv_addr.sin_addr.s_addr = *(u_long *) hent->h_addr_list[0];
				tcp_log_evt_queue = xQueueCreate(MSG_QUEUE_SIZE, 1024);
				if(tcp_log_evt_queue)
					{
					if(xTaskCreate(tcp_log_task, "TCP_log_task", 4096, NULL, USER_TASK_PRIORITY, &tcp_log_task_handle) == pdPASS)
						{
						ret = ESP_OK;
						tcp_log_enable = 1;
						}
					else
						vQueueDelete(tcp_log_evt_queue);
					}
				}
			close(sendsock);
			}
		}
	return ret;
	}
int tcp_log_message(char *message)
	{
	char buf[1024];
	//if !tcp_log_enable try to init first
	if(!tcp_log_enable)
		tcp_log_init();
	if(tcp_log_enable)
		{
		strcpy(buf, USER_MQTT);
		strcat(buf, ":: ");
		strncat(buf, message, 1023 - 2 - strlen(USER_MQTT) );
		xQueueSend(tcp_log_evt_queue, buf, ( TickType_t ) 10);
		}
	return 0;
	}
static void tcp_log_task(void *pvParameters)
	{
	char buf[1024];
	while(1)
		{
		xQueueReceive(tcp_log_evt_queue, buf, portMAX_DELAY);
		int sendsock = socket(AF_INET, SOCK_STREAM, 0);
		if(buf[strlen(buf) - 1] != '\n')
			strcat(buf, "\n");
		int written = 0;
		if (sendsock >= 0)
			{
			if(connect(sendsock, (struct sockaddr *)&srv_addr, sizeof(srv_addr)) >= 0)
				{
				while(written < strlen(buf))
					{
					int sent = send(sendsock, buf + written, strlen(buf), 0);
					if (sent < 0)
						break;
					else
						written += sent;
					}
				}
			close(sendsock);
			}
		}
	}

Simply running “free” command, implemented like this

Code: Select all

static int free_mem(int argc, char **argv)
{
    my_printf("%d", esp_get_free_heap_size());
    return 0;
}

at an interval of a few seconds the output shows lower and lower values. If I wait few minutes and issue the command again the free memory recovers showing again higher value.
If console_state == CONSOLE_ON, which means serial console, there is no impact on memory.
It looks either I’m doing something wrong or it’s something related to memory management.
Any suggestions?

[Horvat]

Maybe your application is producing more log messages, than the what it its capable to send over TCP?

[ves011]

it cannot be the case.
Simply "free" comand outputs just a 5 digit number + prefix, which eats 212 bytes
here is a captured output
wp01:: 21612
wp01:: 21400
wp01:: 21188
wp01:: 20976
wp01:: 20764
wp01:: 20552
wp01:: 20340
wp01:: 20128
wp01:: 19916
wp01:: 19704

[Horvat]

There is a lot of unnecessary data copying going on in your code. First, in tcp_log_message function you write the log message into local variable. The content of the variable, whole 1024 bytes, is then sent into the queue. tcp_log_task function then fetches an item from queue and copies log message into local variable, which is then sent over the network.

You could preallocate a few 1kb buffers and use two queues to pass pointers to these buffers between tcp_log_message and tcp_log_task. If this change is not going solve your issues completely, then at least it should delay them.