MQTT Client causes Chip Hardlock

[EarthAndy]

E.g., this log message
"E (5056) McuCanBusChannel_Esp32: FAILED to Queue TWAI Message for Transmit!"
before the reboot, is that expected to be happening?
If not, I'd first go looking into where/why this happens, checking (error) return values to locate and identify the problem.

The TWAI error message happens when I turn on task watchdog. I have no clue why that is happening but is not the foremost concern. I'm concerned about the MQTT hardlocks.

I'm not in a position to post my entire codebase for my project. What I posted above was meant to just show how I've been mitigating the issue.

Here is the entirety of the MCU layer I have for my MQTT subsystem:

Code: Select all

static void MCU_Default_MQTT5EventRouter(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data);

//==============================================================================
// Class Constructor
//==============================================================================
MCU_MQTT_Client_ESP32::MCU_MQTT_Client_ESP32(void)
{    
    esp_log_level_set( MCU_LOG_Tag, MQTT_MCU_CLIENT_LOG_LEVEL);
    esp_log_level_set( "mqtt5_client", MQTT_ESP32API_CLIENT_LOG_LEVEL);
    esp_log_level_set( "MQTT5EventHandler", MQTT_MCU_CLIENT_LOG_LEVEL);



    ESP_LOGI(MCU_LOG_Tag, "[0x%p] MCU_MQTT_Client_ESP32 Startup..", this);
}


//==============================================================================
// Link User Property Array and Conection Property Structures for use by class.
//==============================================================================
void MCU_MQTT_Client_ESP32::SET_PropertyStructures_ESP32( esp_mqtt5_user_property_item_t UserProperties[], int NumItems,  esp_mqtt_client_config_t *mqtt5_cfg, esp_mqtt5_connection_property_config_t* ConnConfig )
{
    ESP_LOGI(MCU_LOG_Tag, "SET_PropertyStructures_ESP32 [0x%p]", UserProperties);

    UserPropertyArray = UserProperties;
    UserPropertyArrayLength = NumItems;
    ESP_LOGI(MCU_LOG_Tag, "[0x%p] Publish_ESP32: UserPropertyArray=[%p %d]", this, UserPropertyArray, UserPropertyArrayLength );

    MQTT5_ConnectionConfig = mqtt5_cfg;
    MQTT5_ConnectionProperty = ConnConfig;
}


//==============================================================================
// Set Broker Connection Info
//==============================================================================
void MCU_MQTT_Client_ESP32::SET_BrokerConnInfo_ESP32( std::string BrokerURI, std::string UserName, std::string Password)
{
    strcpy( (char*)MQTT5_ConnectionConfig->broker.address.uri, BrokerURI.c_str() );
    strcpy( (char*)MQTT5_ConnectionConfig->credentials.username, UserName.c_str() );
    strcpy( (char*)MQTT5_ConnectionConfig->credentials.authentication.password, Password.c_str());
    MQTT5_ConnectionConfig->network.disable_auto_reconnect = true;  // TODO: Set These externally?
}


//==============================================================================
// Set Last Will Info
// TODO:  Set up Will Topic to use the standard formatting once we have it.
//==============================================================================
void MCU_MQTT_Client_ESP32::SET_LastWillInfo_ESP32( std::string TopicBase, std::string WillMsg )
{
    strcpy( (char*)MQTT5_ConnectionConfig->session.last_will.topic, (TopicBase + "/lastwill").c_str() );
    strcpy( (char*)MQTT5_ConnectionConfig->session.last_will.msg, WillMsg.c_str() );
    MQTT5_ConnectionConfig->session.last_will.msg_len = WillMsg.length();
    MQTT5_ConnectionConfig->session.last_will.qos = 1;  // TODO: Set These externally?
    MQTT5_ConnectionConfig->session.last_will.retain = true;  // TODO: Set These externally?
}


//==============================================================================
// MQTT Client Connection Process
//==============================================================================
esp_mqtt_client_handle_t MCU_MQTT_Client_ESP32::MCU_init(void)
{

#ifdef MAGIC_HARDLOCK_FIX_A
    ESP_LOGV(MCU_LOG_Tag, "esp_mqtt_client_init...");
#endif
    client_handle = esp_mqtt_client_init(MQTT5_ConnectionConfig);

#ifdef MAGIC_HARDLOCK_FIX_B
    ESP_LOGV(MCU_LOG_Tag, "esp_mqtt5_client_set_user_properties...");
#endif
    esp_mqtt5_client_set_user_property(&MQTT5_ConnectionProperty->user_property, UserPropertyArray, UserPropertyArrayLength);
    esp_mqtt5_client_set_user_property(&MQTT5_ConnectionProperty->will_user_property, UserPropertyArray, UserPropertyArrayLength);

#ifdef MAGIC_HARDLOCK_FIX_C
    ESP_LOGV(MCU_LOG_Tag, "esp_mqtt5_client_set_connect_property...");
#endif
    esp_mqtt5_client_set_connect_property(client_handle, MQTT5_ConnectionProperty);

#ifdef MAGIC_HARDLOCK_FIX_D
    ESP_LOGV(MCU_LOG_Tag, "esp_mqtt5_client_delete_user_property...");
#endif
    esp_mqtt5_client_delete_user_property(MQTT5_ConnectionProperty->user_property);
    esp_mqtt5_client_delete_user_property(MQTT5_ConnectionProperty->will_user_property);

#ifdef MAGIC_HARDLOCK_FIX_E
    ESP_LOGV(MCU_LOG_Tag, "esp_mqtt_client_register_event...");
#endif
    esp_mqtt_client_register_event(client_handle, (esp_mqtt_event_id_t)ESP_EVENT_ANY_ID, MCU_Default_MQTT5EventRouter, (void *)this);


#ifdef MAGIC_HARDLOCK_FIX_A
    ESP_LOGD(MCU_LOG_Tag, "MCU_MQTT_Client_ESP32::MCU_init() == %p", client_handle );
#endif

    return client_handle;
}


//==============================================================================
// MQTT Client Connection Process
//==============================================================================
void MCU_MQTT_Client_ESP32::MCU_Connect(void)
{
#ifdef MAGIC_HARDLOCK_FIX_B
    ESP_LOGD(MCU_LOG_Tag, "[%p:%u] Connecting to MQTT Broker... %p", this, __LINE__, client_handle);
#endif

    vTaskDelay(50);

    esp_err_t ret = esp_mqtt_client_start(client_handle);

    if(ret)
        ESP_LOGE(MCU_LOG_Tag, "Could not start MQTT Client!");
    else
        ESP_LOGI(MCU_LOG_Tag, "   MCU_MQTT_Client_ESP32 Connected with Client Handle: [%p]", client_handle);

}


//==============================================================================
// MQTT Client Connection Process
//==============================================================================
void MCU_MQTT_Client_ESP32::MCU_Disonnect(void)
{
#ifdef MAGIC_HARDLOCK_FIX_C
    ESP_LOGD(MCU_LOG_Tag, "Disconnecting from MQTT Broker...");
#endif

    esp_mqtt_client_stop(client_handle);

#ifdef MAGIC_HARDLOCK_FIX_D
    ESP_LOGI(MCU_LOG_Tag, "   MCU_MQTT_Client_ESP32 disconnected.");
#endif

}



//==============================================================================
// MQTT Subscrition Activation
// TRIMMED VERSION - culled out the Subscriber Properties parts
//==============================================================================
void MCU_MQTT_Client_ESP32::Subscribe_ESP32(void* MCUClientHandle, char* Topic, int QoS )
{
#ifdef MAGIC_HARDLOCK_FIX_E
    ESP_LOGD(MCU_LOG_Tag, "[0x%p] MCU_MQTT_PubSub_ESP32 Subscribing...", this);
#endif

    esp_mqtt_client_handle_t tClient = (esp_mqtt_client_handle_t)MCUClientHandle;

#ifdef MAGIC_HARDLOCK_FIX_A
    ESP_LOGV(MCU_LOG_Tag, "     esp_mqtt_client_subscribe...");
#endif
    int msg_id = esp_mqtt_client_subscribe(tClient, Topic, QoS);

#ifdef MAGIC_HARDLOCK_FIX_B
    ESP_LOGD(MCU_LOG_Tag, ESC_FGCOLOR_BRIGHT_YELLOW "[0x%p]     MCU_MQTT_PubSub_ESP32 Subscribed! (msg_id=%d)", this, msg_id);
#endif

}


//==============================================================================
// MQTT Publish Topic
//==============================================================================
void MCU_MQTT_Client_ESP32::Publish_ESP32( void* MCUClientHandle, char* Topic, char* Message, int QoS, int RetainTopic )
{
#ifdef MAGIC_HARDLOCK_FIX_C
    ESP_LOGV(MCU_LOG_Tag, "[0x%p] MCU_MQTT_PubSub_ESP32 Publishing...[%s]", this, Topic);

    ESP_LOGV(MCU_LOG_Tag, "[0x%p]     Publish_ESP32: UserPropertyArray=[%p %d]", this, UserPropertyArray, UserPropertyArrayLength );
    ESP_LOGV(MCU_LOG_Tag, "[0x%p]     Publish_ESP32: MCUClientHandle=[%p]", this, MCUClientHandle);
    ESP_LOGV(MCU_LOG_Tag, "[0x%p]     Publish_ESP32: PubTopic=%s", this, Topic );
    ESP_LOGV(MCU_LOG_Tag, "[0x%p]     Publish_ESP32: PubMessage=[%s]", this, Message );
    ESP_LOGV(MCU_LOG_Tag, "[0x%p]     Publish_ESP32: PubQoS=[%d]", this, QoS );
#endif

    esp_mqtt_client_handle_t tClient = (esp_mqtt_client_handle_t)MCUClientHandle;

#ifdef MAGIC_HARDLOCK_FIX_D
    ESP_LOGV(MCU_LOG_Tag, "     esp_mqtt_client_publish... [%p]", tClient);
#endif

#ifndef MQTT_MCU_NPUBLISH_ENQUEUE
    int msg_id = esp_mqtt_client_publish(tClient, Topic, Message, 0, QoS, RetainTopic);
#else
    int msg_id = esp_mqtt_client_enqueue(tClient, Topic, Message, 0, QoS, RetainTopic, true);
#endif

#ifdef MAGIC_HARDLOCK_FIX_E
    ESP_LOGD(MCU_LOG_Tag, ESC_BGCOLOR_BRIGHT_WHITE ESC_FGCOLOR_BRIGHT_BLUE "[0x%p]     MCU_MQTT_PubSub_ESP32 Published! (msg_id=%d) [%s]" ESC_FGCOLOR_DEFAULT ESC_BGCOLOR_DEFAULT "\n\n" , this, msg_id, Topic);
#endif
    msg_id = msg_id + 1;

}


//==============================================================================
//  Get the User Propery Array pointer
//==============================================================================
esp_mqtt5_user_property_item_t* MCU_MQTT_Client_ESP32::GET_UserPropertyArray(void)
{
    return UserPropertyArray;
}


//==============================================================================
//  Get the User Propery Array Length
//==============================================================================
int MCU_MQTT_Client_ESP32::GET_UserPropertyArrayLength(void)
{
    return UserPropertyArrayLength;
}


//==============================================================================
//  Overridable Event Router
//==============================================================================
void MCU_MQTT_Client_ESP32::EventRouter(void)
{
    ESP_LOGE(MCU_LOG_Tag, "MCU_MQTT_Client_ESP32 Default EventRouter  EventID=%ld  EventData=%p", LastEventID, LastEventData);
}


//==============================================================================
// HandleEvent_ESP32
// This function stores incoming MQTT Events and notifies overridable
//   event router for event processing.
//==============================================================================
void MCU_MQTT_Client_ESP32::HandleEvent_ESP32(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data)
{
        LastEventData = event_data;
        LastEventID = event_id;
        LastEventReady = true;
        EventRouter();
}


//==============================================================================
// Static Non-Class Event Router
//==============================================================================
static void MCU_Default_MQTT5EventRouter(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data)
{
    MCU_MQTT_Client_ESP32 *oClient = (MCU_MQTT_Client_ESP32 *)handler_args;

//    ESP_LOGD("MQTT5EventHandler", "[DEFAULT] [0x%p] Event dispatched from event loop base=%s, event_id=%" PRIi32, oClient, base, event_id);
#ifdef MAGIC_HARDLOCK_FIX_A
    ESP_LOGD("MQTT5EventHandler", "[DEFAULT] [0x%p] Event dispatched from event loop base=%s, event_id=%", oClient, base, event_id);
#endif

    oClient->HandleEvent_ESP32(handler_args, base, event_id, event_data);
}

Code: Select all

esp_mqtt5_user_property_item_t esp32_mqtt_user_property_array[] = {
        {"board", "esp32"},
        {"u", "user"},
        {"p", "password"}
    };

esp_mqtt_client_config_t esp32_mqtt_connection_config = {
//        .broker.address.uri = CONFIG_BROKER_URL,
//    .broker.address.uri = "mqtts://mqtt-emqx.earthfuneral.tech",
    .broker.address.uri = "mqtts://example.com",
    .session.protocol_ver = MQTT_PROTOCOL_V_5,
    .network.disable_auto_reconnect = true,
    .credentials.username = "",
    .credentials.authentication.password = "",
    .session.last_will.topic = "/topic/will",
    .session.last_will.msg = "i will leave",
    .session.last_will.msg_len = 12,
    .session.last_will.qos = 1,
    .session.last_will.retain = true,
};

esp_mqtt5_connection_property_config_t esp32_mqtt_connection_property = {
    .session_expiry_interval = 10,
    .maximum_packet_size = 1024,
    .receive_maximum = 65535,
    .topic_alias_maximum = 2,
    .request_resp_info = true,
    .request_problem_info = true,
    .will_delay_interval = 10,
    .payload_format_indicator = true,
    .message_expiry_interval = 10,
    .response_topic = "/test/response",
    .correlation_data = "",
    .correlation_data_len = 0,
};

This should be recognizable as basically the ESP-IDF example code from the API (as I mentioned above), just wrapped in a class for integration into the rest of my system. Any other parts of the system that interface with MQTT run interfaces through a single instance of this class. In my system I have a FreeRTOS task that is spawned off that handles all MQTT, it constructs and initializes this class, initiates the connection via this class, and manages all publish messages via this class.


Hardlocks occur at the exact same spot for each build (but often different between builds depending on which ESP_LOG lines are active), but it is always when MQTT Connects, Subscribes, Publishes, or Receives using the functions in the code block provided here.


Thanks for looking.

[EarthAndy]

I was able to solve my issue by moving some stuff like .bss to the SPIRAM. I did also some changes in the menuconfig, I can post them in a few days, right now I am traveling.
But you don’t have SPIRAM, right? Well, maybe it will help somehow.

Yeah, this is kind of the path that I'm going down - it seems to be caused by code or ram alignment. I am not using external ram, so your solution won't directly help, but does go to second my thoughts that this is a ram issue.

A

[MicroController]

The TWAI error message happens when I turn on task watchdog. I have no clue why that is happening but is not the foremost concern.

Ok. I'm just wondering if whatever causes the problem with the TWAI messages may also be breaking your MQTT... like lack of heap, for instance.

it seems to be caused by code or ram alignment.

Then there's a bug in your code, maybe a race condition, or lack of heap or stack space.