ESP32 UART FIFO Operation

[berlinetta]

Hello All,

I have been developing with ESP-IDF v4.3.2. Under this version, I have UART communications working flawlessly over both UART0 and UART2.

I recently attempted to upgrade the project to utilize the more recent releases of ESP-IDF. I discovered that the UART receive FIFO operation is no longer working as expected. While the transmit function is working well (and allowing me to dump diagnostic information) the receive operations are fouled up. After much testing of the project builds under different versions of the IDF, I discovered that this apparently crept in with v4.4.x.

My testing focused on UART0 operation and included the use of several dumps of the GPIO and UART configuration registers to determine if I could detect any differences in the configuration of the hardware between v4.3.2 and v4.4.1. I am attempting to utilize UART0 with its default connections for TxD and RxD signals (GPIO1 and GPIO3). I noticed the following changes in the GPIO configuration register values between the two builds:

Code: Select all

GPIO_ENABLE_REG:                    v4.3.1: 0x0E620FB6    v4.4.1: 0x0E620FB4    (Set GPIO1 as output vs. set GPIO1 as input)
GPIO_FUNC14_IN_SEL_CFG_REG:         v4.3.1: 0x00000083    v4.4.1: 0x00000030    (bypass GPIO matrix, connect U0RxD to GPIO3)
GPIO_FUNC1_OUT_SEL_CFG_REG:         v4.3.1: 0x0000000E    v4.4.1: 0x00000100    (routing U0TxD to GPIO1 vs. routing bit 1 of GPIO_OUT_REG to GPIO1)
IO_MUX_GPIO1_REG:                   v4.3.1: 0x00002A00    v4.4.1: 0x00000A00    (routing TxD0 through the matrix vs. bypassing the matrix)
IO_MUX_GPIO3_REG:                   v4.3.1: 0x00002B00    v4.4.1: 0x00000B00    (routing RxD0 through the matrix vs. bypassing the matrix)

Obviously the discrepancies in routing the U0TxD to the GPIO1 output pin do not pose any issues, as the transmitter is producing traffic just fine.

The comparison of the UART configuration registers did not yield any significant differences, with the exception that the MEM_RX_STATUS register "rd_addr" parameter was non-zero on the nonworking build under v4.4.1 (register offset 0x60). Here are the dumps of the full register values for each version:

v4.3.1:
04:00002020 08:00000000 0C:00004113 10:00000000 14:009000AD 18:00001000 1C:6678C000 20:0800001C 24:82006470 28:000FFFFF 2C:000FFFFF 30:00000000 34:00000000 38:000000F0 3C:1311E000 40:00A00100 44:00000000 48:00186A00 4C:00186A00 50:00001E00 54:0000032B 58:00000088 5C:00034010 60:00300600 64:00000000 68:000FFFFF 6C:000FFFFF 70:00000000 74:00000000 78:15122500 7C:00000500

v4.4.1:
04:00002020 08:00000000 0C:00004113 10:00000000 14:009000AD 18:00001000 1C:6576C000 20:0800001C 24:82006470 28:000FFFFF 2C:000FFFFF 30:00000000 34:00000000 38:000000F0 3C:1311E000 40:00A00100 44:00000000 48:00186A00 4C:00186A00 50:00001E00 54:0000032B 58:00000088 5C:00038010 60:00300610 64:00000000 68:000FFFFF 6C:000FFFFF 70:00000000 74:00000000 78:15122500 7C:00000500

Finally, I instrumented my interrupt handler to dump some information about the number of characters available in the FIFO before and after I read each available character from the FIFO. I tested this operation with single character transfers and noticed that the rd_addr parameter is incrementing by '1' on each transfer when built with v4.3.1, but the value increments by more than '1' when built under v4.4.1.

The number of bytes available in the FIFO is calculated as follows in the interrupt handler:

Code: Select all

uint16_t u16NumBytesAvail = ( UART0.mem_cnt_status.rx_cnt << 8 ) + UART0.status.rxfifo_cnt;

The instrumented code dumps the calculated u16NumBytesAvail value plus the value of [UART0.mem_cnt_status.rx_cnt << 8] and the value of [UART0.status.rxfifo_cnt] before and after each character is read from the FIFO. The code also dumps the values of the rd_addr and wr_addr parameters before and after the character is read from the FIFO. The test results are as follows:

Code: Select all

                                           expected             v4.3.1 build:           v4.4.1 build:
u16NumBytesAvail:                           1                        1                       1
UART0.mem_cnt_status.rx_cnt << 8:           0                        0                       0
UART0.status.rxfifo_cnt:                    1                        1                       0*
UART0.mem_rx_status.rd_addr:                384                      384                     474*
UART0.mem_rx_status.wr_addr:                385                      385                     385 
Hexadecimal value read from the FIFO:       0x31                     0x31                    (random)*
UART0.mem_cnt_status.rx_cnt << 8:           0                        0                       0
UART0.status.rxfifo_cnt:                    0                        0                       0
UART0.mem_rx_status.rd_addr:                385                      385                     501*
UART0.mem_rx_status.wr_addr:                385                      385                     385

I am presuming this is something occurring within the silicon, as the UART0 register set refers to the physical peripheral registers, correct?

Can anyone offer any explanation for this strange behavior of the receive FIFO operation?

Note that I had a thought that perhaps some other driver was loaded and attempting to pull a crazy number of characters from the FIFO before and or after my interrupt handler. I attempted to execute the uart_driver_delete() call prior to calling esp_intr_alloc() to map my handler as below...

Code: Select all

uart_driver_delete( UART_NUM_0 );
ret_val = esp_intr_alloc( ETS_UART0_INTR_SOURCE, flags, UartInterruptHandler, &Uart0Params, &Uart0Params.intr_handle );

Adding this driver delete call had no effect on the erroneous operation of the Rx FIFO.

Best Regards,
Mark

[MicroController]

Code: Select all

UART0.status.rxfifo_cnt:                    1                        1                       0*

So you're receiving a UART_RXFIFO_FULL_INT but rxfifo_cnt reads as 0?

The bug would be much easier to find if you'd provide the code to reproduce the issue.

[berlinetta]

Hello MicroController... Thanks for the quick reply!

I am enclosing the code for the ISR below... were you in need of anything else?

FYI, the UART_PARAMS structure is something I have created for managing the connection - I've included that structure definition as well.

Best Regards,
Mark

Code: Select all

/****************************************************************************
*  @brief UART software device instance structure.
*
* Used to retain software state information of an associated hardware module instance.
*
* NOTE: The fields of this structure should not be altered by the user
*      application; they are reserved for module-internal use only.
****************************************************************************/
typedef struct
{
    /* UART Transmit buffer */
    volatile uint8_t* pu8TxBuffer;
    /* UART Transmit buffer size */
    volatile uint16_t u16TxBufferSize;
    /* Index into Tx buffer of next byte to write */
    volatile uint16_t u16TxBuffWriteIndex;
    /* Index into TX buffer of next byte to read */
    volatile uint16_t u16TxBuffReadIndex;
    /* UART Receive buffer */
    volatile uint8_t* pu8RxBuffer;
    /* UART Receive buffer size */
    volatile uint16_t u16RxBufferSize;
    /* Index into Rx buffer of next byte to write */
    volatile uint16_t u16RxBuffWriteIndex;
    /* Index into RX buffer of next byte to read */
    volatile uint16_t u16RxBuffReadIndex;
    /* Pointer to physical UART register set */
    volatile uart_dev_t* pUartHw;
    /* Pointer to interrupt handle */
    intr_handle_t intr_handle;
    /* Configuration flags */
    bool bRxInterruptDriven;
    bool bTxInterruptDriven;
    /* Fault Flags */
    UART_FAULT_FLAGS FaultFlags;
} UART_PARAMS;


/**************************************************************************
* UsartInterruptHandler
*//**
*   @Brief      Handles interrupts as they occur
*
*   @param[in]  arg         Pointer to parameter instance
*   @returns    None
**************************************************************************/
static void IRAM_ATTR UartInterruptHandler( void* arg )
{
    UART_PARAMS* pUartParams = (UART_PARAMS*)arg;
    uart_dev_t* pUartHw = pUartParams->pUartHw;

    /* Check for received data FIRST */
    if( pUartParams->bRxInterruptDriven )
    {
        /* Check for reception of any data */
        if( pUartHw->int_st.rxfifo_ovf )
        {
            /* We have encountered an RX overflow... data is corrupted, set the flag */
            pUartParams->FaultFlags.flag.bRxFifoOverflow = true;
            /* Flush the hardware FIFO to prevent re-entry */
            while( pUartHw->mem_cnt_status.rx_cnt )
            {
                (void)pUartHw->fifo.rw_byte;
            }
            /* Clear the interrupt */
            pUartHw->int_clr.rxfifo_ovf = 1u;
        }
        else if( ( pUartHw->int_st.rxfifo_tout ) || ( pUartHw->int_st.rxfifo_full ) )
        {
            /* RX FIFO has data available... check to see if we have encountered any faults */
            /* NOTE: Reception of BREAK places a byte value of zero in the FIFO and sets the framing error flag */
            if( pUartHw->int_st.frm_err )
            {
                /* We have a framing error... data is corrupted, set the flag */
                pUartParams->FaultFlags.flag.bFramingError = true;
                /* Clear the interrupt */
                pUartHw->int_clr.frm_err = 1u;
            }
            else
            {
                /* Otherwise normal data reception */
            }

            /* RX FIFO has data available... read it out */
            uint16_t u16RxBufferSpace = CalculateBufferSpace( pUartParams->u16RxBufferSize, pUartParams->u16RxBuffReadIndex, pUartParams->u16RxBuffWriteIndex );
            uint16_t u16NumBytesAvail = pUartHw->mem_cnt_status.rx_cnt;
            u16NumBytesAvail <<= 8;
            u16NumBytesAvail += pUartHw->status.rxfifo_cnt;

            /* Save what we can based on buffer space */
            while( u16NumBytesAvail && u16RxBufferSpace )
            {
/* Dump debug info before popping FIFO if this interrupt is for UART0 */
if( pUartParams->pUartHw == &UART0 )
{
    DebugPutString( "\r\n>(", DEBUG_PORT_STD );
    DebugPutDoubleWordValue( u16NumBytesAvail, DEBUG_PORT_STD );
    DebugPutString( "=", DEBUG_PORT_STD );
    DebugPutDoubleWordValue( ( pUartHw->mem_cnt_status.rx_cnt << 8 ), DEBUG_PORT_STD );
    DebugPutString( "+", DEBUG_PORT_STD );
    DebugPutDoubleWordValue( pUartHw->status.rxfifo_cnt, DEBUG_PORT_STD );
    DebugPutString( ", pRd=", DEBUG_PORT_STD );
    DebugPutDoubleWordValue( pUartHw->mem_rx_status.rd_addr, DEBUG_PORT_STD );
    DebugPutString( ", pWr=", DEBUG_PORT_STD );
    DebugPutDoubleWordValue( pUartHw->mem_rx_status.wr_addr, DEBUG_PORT_STD );
    DebugPutString( ")", DEBUG_PORT_STD );
}
                #if 1
                *( pUartParams->pu8RxBuffer + pUartParams->u16RxBuffWriteIndex ) = pUartHw->fifo.rw_byte;
                #else
                uint32_t fifo_addr = UART_FIFO_AHB_REG(0);
                *( pUartParams->pu8RxBuffer + pUartParams->u16RxBuffWriteIndex ) = READ_PERI_REG( fifo_addr );
                #endif

/* Dump debug info after popping FIFO if this interrupt is for UART0 */
if( pUartParams->pUartHw == &UART0 )
{
    DebugPutHexChar( *( pUartParams->pu8RxBuffer + pUartParams->u16RxBuffWriteIndex ), DEBUG_PORT_STD );
    DebugPutString( ", ", DEBUG_PORT_STD );
    DebugPutDoubleWordValue( ( pUartHw->mem_cnt_status.rx_cnt << 8 ), DEBUG_PORT_STD );
    DebugPutString( "+", DEBUG_PORT_STD );
    DebugPutDoubleWordValue( pUartHw->status.rxfifo_cnt, DEBUG_PORT_STD );
    DebugPutString( ", pRd=", DEBUG_PORT_STD );
    DebugPutDoubleWordValue( pUartHw->mem_rx_status.rd_addr, DEBUG_PORT_STD );
    DebugPutString( ", pWr=", DEBUG_PORT_STD );
    DebugPutDoubleWordValue( pUartHw->mem_rx_status.wr_addr, DEBUG_PORT_STD );
    DebugPutString( "\r\n", DEBUG_PORT_STD );
}
                u16RxBufferSpace--;
                u16NumBytesAvail--;
                pUartParams->u16RxBuffWriteIndex++;

                if( pUartParams->u16RxBuffWriteIndex >= pUartParams->u16RxBufferSize )
                {
                    pUartParams->u16RxBuffWriteIndex = 0;
                }
            }

            if( u16NumBytesAvail )
            {
                /* We do not have the room in the buffer to save all the data! */
                /* IMPORTANT: Empty the FIFO to prevent re-entry into this handler */
                while( u16NumBytesAvail )
                {
                    (void)pUartHw->fifo.rw_byte;
                    u16NumBytesAvail--;
                }

                /* Set the overflow fault flag */
                pUartParams->FaultFlags.flag.bRxFifoOverflow = true;
            }

            /* Clear the interrupt(s) */
            pUartHw->int_clr.rxfifo_tout = 1u;
            pUartHw->int_clr.rxfifo_full = 1u;
        }
    }

    if( pUartParams->bTxInterruptDriven )
    {
        /* Check for TX FIFO space */
        if( ( pUartHw->int_st.txfifo_empty ) || ( pUartHw->int_st.tx_done ) )
        {
            /* Data FIFO is ready for new data */
            if( ( pUartParams->u16TxBuffReadIndex == pUartParams->u16TxBuffWriteIndex ) )
            {
                /* No more data to send... */
                /* IMPORTANT: Disable the TX_EMPTY interrupt to prevent interrupt flood */
                pUartHw->int_ena.txfifo_empty = 0u;
            }
            else
            {
                /* Fill the TX FIFO as long as we have data and there is room in the FIFO */
                while( ( pUartHw->status.txfifo_cnt < UART_TX_MAX_FIFO_COUNT ) &&
                    ( pUartParams->u16TxBuffReadIndex != pUartParams->u16TxBuffWriteIndex ) )
                {
                    /* Send the next character to the FIFO, increment the read index */
                    pUartHw->fifo.rw_byte = *( pUartParams->pu8TxBuffer + pUartParams->u16TxBuffReadIndex );
                    pUartParams->u16TxBuffReadIndex++;

                    if( pUartParams->u16TxBuffReadIndex >= pUartParams->u16TxBufferSize )   /* Roll over the index if it is larger than the buffer */
                    {
                        pUartParams->u16TxBuffReadIndex = 0;
                    }
                }
            }

            /* Clear the interrupt(s) */
            pUartHw->int_clr.txfifo_empty = 1u;
            pUartHw->int_clr.tx_done = 1u;
        }
    }
}

[berlinetta]

BTW,

The "code" is a rather large project... I will do my best to reduce things to the lowest common denominator and see if I can get a simple project that exhibits the same behavior.

In the interim, below is a sample of what the current code is producing when built with v4.4.1 as I attempt to type single characters into the terminal emulator which is connected to UART0. Note that the code includes a diagnostic command handler which monitors for command requests on UART0. Each character is echoed back to the port as it is received. Since the problem code is not able to read valid characters on the input, the diagnostic code reports the detected command attempts as "Invalid" for each of these entry scenarios.

The first set of responses is produced when the ENTER key is pressed and the terminal emulator is configured to transmit carriage return and line feed (two characters sent).

The second response is generated with the attempt to send a single decimal digit.

The third response is generated when an attempt to enter several key presses in succession - all are single digit numbers.

Notice that the FIFO write pointer values appear to be legitimate and are incrementing by one with each character detected. The FIFO read pointer, however, is incrementing by an unexpected value. And as stated before, the value read from the status.rxfifo_cnt is reported as zero when printed - despite the fact that it must have been non-zero to initially trigger the interrupt, and to calculate the value for u16NumBytesAvail which is printed as a non-zero number. The problematic build is acting as if the status.rxfifo_cnt is cleared after being read, or some other code in the background is popping the FIFO.

Best Regards,
Mark

Code: Select all

>(2=0+0, pRd=424, pWr=386)E0, 0+0, pRd=451, pWr=386

>(1=0+0, pRd=480, pWr=386)B6, 0+0, pRd=507, pWr=386
à¶
Invalid

>(1=0+0, pRd=421, pWr=387)F9, 0+0, pRd=448, pWr=387
ù
Invalid

>(1=0+0, pRd=489, pWr=388)17, 0+0, pRd=388, pWr=388

>(1=0+0, pRd=418, pWr=389)3A, 0+0, pRd=445, pWr=389
:
>(1=0+0, pRd=475, pWr=390)D9, 0+0, pRd=502, pWr=390
Ù
>(1=0+0, pRd=404, pWr=391)45, 0+0, pRd=431, pWr=391
E
>(1=0+0, pRd=461, pWr=392)84, 0+0, pRd=488, pWr=392

>(1=0+0, pRd=390, pWr=393)AF, 0+0, pRd=417, pWr=393
¯
>(1=0+0, pRd=447, pWr=394)9B, 0+0, pRd=474, pWr=394

(1=0+0, pRd=504, pWr=395)33, 0+0, pRd=403, pWr=395
3
Invalid

[berlinetta]

Hello MicroController,

I have managed to reduce the project to the initialization code, peripheral drivers and diagnostic support. This project still exhibits the problem described in my original post. Note that I am developing under VSCode with the PlatformIO extension (Core v6.1.11, Home v3.4.4). The PlatformIO project can be retrieved from my DropBox location using the following link:

https://www.dropbox.com/scl/fi/46mu1y9o ... 0kjmz&dl=0

Please let me know if you have any problems reproducing this behavior under ESP-IDF v4.4.1 (PlatformIO Espressif32 platform v5.0.0) or higher.

Best Regards,
Mark

[MicroController]

I wouldn't focus too much on the RX-FIFO pointer having an "unexpected" value after software tried to read from the empty FIFO.
Any chance that during your testing some code crept in which causes a 'double-read' of the data register?

[berlinetta]

MicroController,

Reading from an empty FIFO would obviously result in the read pointer getting corrupted. No, there is no chance that there were any changes in MY code where the FIFO would be getting read an extra time. Please keep in mind... the SAME code I have will work flawlessly on any ESP-IDF prior to v4.4.x. Therefore, in my mind this says there must be something different in the ESP-IDF that is causing this issue. I am trying to understand what change may have caused this behavior.

If you are able to utilize the example project I provided which replicates this issue, you can witness this behavior for yourself. An adjustment to the platformio.ini file can be done to specify the platform as espressif32@5.0.0 or greater (utilizing ESP-IDF v4.4.1 or greater). This action results in the FIFO problems described in this thread. If the platform is set at espressif32@4.4.0 or lower (utilizing ESP-IDF v4.3.2 or lower) the UART works just fine and the FIFO operates as expected.

The documentation is not the greatest, but in all my research it appears that the parameters of the uart_dev_t structure are literally hardware registers in the silicon of the UART peripheral. Therefore, these read and write pointers for the Rx FIFO should only change if the FIFO is accessed through the fifo.rw_byte parameter of the structure. I struggle to comprehend what could be going on with the change in ESP-IDF code that could adversely affect the FIFO behavior.

Is my assessment incorrect? Or is the UART FIFO management software controlled, where another module of code is responsible for handling the uart_dev_t structure parameters?

I don't exactly know what the ESP-IDF code is doing when accessing UART0 for the purposes of transmitting diagnostic data. As another test, I attempted to call the uart_driver_delete() function in an attempt to remove any other driver activity from the equation, but it had no effect on this issue.

A comparison of the 4.3.2 and 4.4.1 releases does not yield any significant changes to the uart.c module that might influence this FIFO behavior. Is there another module of code that may have changed to effect the FIFO operations?

Best Regards,
Mark

[MicroController]

I struggle to comprehend what could be going on with the change in ESP-IDF code that could adversely affect the FIFO behavior.

The custom UART driver/ISR may obviously compete with the IDF's UART driver, and with the bootloader's and application's use of UART0 for logging if enabled.
Of course, I don't know where and how this conflict occurs since I haven't studied the 60+kb of custom UART driver code you provided, but I assume the problem would be in the way the 'takeover' of the hardware from the IDF (bootloader,...) by the custom driver is done.

[berlinetta]

Hello MicroController,

The bootloader should no longer be in the picture once it boots the application code. Yes, I was originally concerned about a conflict between print statements in supporting IDF code modules and my application code, but things have been working without issue until more recent versions of the IDF. I did note that I had attempted the use of the uart_driver_delete() call before registering my interrupt handler in an attempt to circumvent any issues with that contention - but to no avail.

I have been spending countless days examining this issue and characterizing the behavior to narrow down the root cause. I have finally proven that this issue is somehow related to a change in the ESP-IDF - and even narrowed it down to the version that this crept in on. I spent more time analyzing differences in the releases to determine if there were any obvious red flags, but have not found any smoking gun. Documentation is very poor and I do not have the "insider knowledge" about the design of the Espressif code to be able to make educated decisions on where to look next. As a last resort, I have looked to the forum for some answers.

Please keep in mind, the very same application code does not exhibit any issues under the older versions of the ESP-IDF.

I have taken another day to provide a simplified project which replicates the problematic behavior only under newer versions of the IDF. While the esp_uart code module may be 60K in size, only the portions relative to UART0 are of interest in this scenario. The initialization of the UART can be found in the Uart0Init() function. The UartInterruptHandler() can be examined for any issues too.

Would it be possible for you to build my example project and verify the behavior? Perhaps you could then help me track something down / determine an effective work-around?

Best Regards,
Mark

[berlinetta]

Hello Microcontroller,

I have been examining the operation of the interrupt configuration in my simplified test project, and it appears that my code is properly configuring the interrupt matrix in such a way that my UART0 interrupt handler is the sole owner of the peripheral interrupt (not shared). Analysis of the interrupt vector initialization shows the UART0 interrupt is routed to interrupt 0x13 (19) on the core0 processor and the vector table is holding the address of my interrupt handler and the pointer to the argument I have specified. This should ensure my handler is the only one invoked when the interrupt fires.

BTW - I am configured for single-core operation, so the vector table for core1 is showing all vectors are routed to the xt_unhandled_interrupt() routine and all arguments are set to their respective interrupt number (default state for the table).

I am beginning to suspect there is a new "feature" implemented in ESP_IDF versions 4.4.x and onward which is polling the UART0 register set to look for received characters, and is automatically "popping" a specific number of characters from the FIFO regardless of what the FIFO contains per the read pointer. Perhaps there is some new code that expects to receive commands of a specific packet size on UART0? Could this be the case?

Best Regards,
Mark