RTOS running on one core only

[HelWeb]

A picture tells more than 1000 words

[chrismerck]

@HelWeb:

Just want to thank you for posting your technique for running without RTOS on Core1. This is extremely useful information to my team here to work around interrupt performance issues on Core0. We will let you know our results!

Merci vielmal!

[apuder]

Thanks for the nice explanations. I have been trying the same thing. I have a question about portDISABLE_INTERRUPTS(): does this turn off interrupts only on the core where that function is called or turn off interrupts for both cores? I am calling this function on core 1, but then I noticed some unusual behavior on core 0 (more precisely, on core 0 I try to connect to WiFi but nothing happens (with "nothing" I mean no error, no log; as if I didn't call any WiFi API))
TIA,
AP

[apuder]

followup to my own question. portDISABLE_INTERRUPTS() only disables interrupts on the core where that function is called. So in the scenario discussed here, interrupts on core 0 are still enabled. However, I have noticed that FreeRTOS will create an "ipc" task on core 1. I wonder if disabling interrupts on core 1 might have impact on IPC on core 0. That could explain the behavior I observe. Anyone out there has an idea?
AP

[apuder]

more digging: the problem seems to be indeed related to IPC. I logged esp_ipc_call() for core 1 and I was surprised how much code is executed on that core. E.g., calling nvs_flash_init() will trigger esp_ipc_call() on core 1. If interrupts are disabled on core 1 at that moment, then the ESP32 will just hang.

Anyone out there know why so much code is executed on core 1 via esp_ipc_call()? Could this be related to the interrupt that is used to synchronize D-port accesses?

AP

[chrismerck]

Hey AP, in fact, we have run into a similar issue here.

If you read in the ESP-IDF version of FreeRTOS code, you will see conditional compilation in many places like this:

Code: Select all

#if !CONFIG_FREERTOS_UNICORE

You will often see while loops used to spinlock waiting for the other core to release some resource. It is unclear to me at this time exactly what all the shared resources are, but it is definitely related to flash access.

The problem we have is that we need to read/write a flash-based database on Core 0 whilst we are performing timing-sensitive RF work on Core 1 (12uS deadlines). We tried adding the following to the Core 1 loop:

Code: Select all

      portENABLE_INTERRUPTS();
      ets_delay_us(1);
      portDISABLE_INTERRUPTS();

Which relieves the deadlock, but unfortunately introduces >12uS of latency and breaks our RF requirements.

We are currently investigating how to use the UNICORE mode of FreeRTOS and manually launch our small RF code on Core 1 -- a major departure from HelWeb's method.

[apuder]

The synchronization between the cores is definitely related to NVS access. I saw that when the ESP performs a core dump, it can write to NVS *without* syncing with the second core. In another thread I asked the question if I could use the same trick:
https://esp32.com/viewtopic.php?f=2&t=10901

Someone replied by saying probably not. I haven't tried it yet. My solution is to have a global volatile variable where core 0 sets a bit to signal to core 1 that it should temporarily enable interrupts. Core 0 does this just before it wants to access NVS. Part of the busy-wait loop on core 1 I also check this volatile variable. Most of the time I can keep interrupts turned off on core 1 and only for brief moments when I need to access NVS on core 0 I turn on interrupts. This is a little better than what you do.
AP

[chrismerck]

only for brief moments

Thanks for the quick reply!

The minimum duration of those brief moments is dictated by the design of ESP-IDF. I'm not sure what Espressif's design latency is, but empirical testing on our end makes it seem greater than our requirement of <12uS, because we are missing deadlines when we enable interrupts even for just 1 uS of busywait.

[apuder]

Not sure I understand what you are saying. Did you disable the watchdog timer on core 1? With that I can keep interrupts on core 1 turned off indefinitely as long as core 0 is not using the NVS. What I was proposing with the volatile global variable is just a simple signaling mechanism where I can temporarily enable interrupts again on core 1. The code sniplet you posted in your original message seems to suggest that you disable and enable interrupts with each iteration of your busywait loop on core 1. That is not necessary.
AP

[chrismerck]

temporarily enable interrupts again on core 1

At which time you introduce potential delays into the code executing on core 1.

The code sniplet you posted in your original message seems to suggest that you disable and enable interrupts with each iteration of your busywait loop on core 1.

Our requirement is that we have a deadline every 25uS, continuously---we are servicing I/O pins based on an external 40kHz clock. Any missed deadline results in a corrupted transmit/receive signal. We we need to run about 13uS of code each deadline, so that leaves 12uS of time left to potentially service interrupts between deadlines. There's never a time to take a breath and accept a longer interruption.

We clearly didn't realize that the ESP-IDF design requires interrupts to be serviced by core 1 to permit core 0 operations. A DMA peripheral would be preferable, but RMT cannot sync to the external 40kHz bit clock, and the SPI driver doesn't seem to support the use case either. But we may need to revisit the SPI possibility.