IPC Interfering with RMT
Posted: Fri Dec 10, 2021 11:01 pm
Project description and problem
I'm working on a project that implements a file-based web server to control a string of WS2812 LEDs. The web server by itself works great, and the LEDs by themselves work great. The problem is when I have both running simultaneously, the LEDs flash random colors any time a client gets a file from the web server. Here's some details about the LEDs and the web server:
LEDs - There's a chain of 114 WS2812 LEDs connected to pin 23, which are driven by the RMT module. There is a logic level converter before the first LED to convert the 3.3V signal to 5V for the LEDs. My code is fairly similar to the led_strip example in the IDF. I've created a dedicated LED task pinned to core 1 with max priority, everything else has been configured to run on core 0 (part of my debugging process, see debugging section below).
Web server - It's a normal HTTP server that sends files to HTTP clients using SPIFFS. Right now it's only sending a single text file with "Hello world" as its contents, so nothing intense. My code is fairly similar to the file_serving example in the IDF. This runs on core 0 along with everything else, except the LED task.
My diagnosis, workaround, and questions
After lots of digging through code (see debugging section below), I believe the problem has to do with the IPC feature interrupting the RMT driver. If I temporarily suspend the IPC task while the RMT module is sending data, there's no more flashing of random colors, everything works great. Here's a snippet of my code:
I don't really know much about the IPC tasks and how they're used. I've skimmed through the IPC API reference (https://docs.espressif.com/projects/esp ... m/ipc.html), however it doesn't say much on what actually uses it. The only info I've found about that is from this GitHub issue from 3 years ago (https://github.com/espressif/esp-idf/issues/1482), saying that the IPC tasks are used during flash operations (eg. SPIFFS file reading, like my web server does). Can anyone provide info on how the IPC is actually used? It's probably not super important to know, but I am curious! It also seems to me that the IPC shouldn't be necessary if I've got every task pinned to core 0 except my LED task, since no data is being transferred between the cores (unless I'm mistaken?). Is there a way to disable it? Or to prevent SPIFFS from using it?
Suspending the IPC task like this seems to work well, but I suspect it's not recommended, and I haven't tested it thoroughly enough to feel confident in using it. There are very few config options for the IPC tasks, which leads me to believe that either 1) it's an essential thing that shouldn't be touched, or 2) it's something used so rarely that it's never been polished with more config options. Both sound plausible to me with my current knowledge of what the IPC tasks do (which is admittedly not much). Does anyone know?
Is this a bug? I feel like the RMT module shouldn't be interrupted under most circumstances, some things have very tight timing requirements (eg. WS2812 LEDs), and simply reading a file shouldn't interrupt that. Should I create an issue in the IDF GitHub? I would like to limit how many workaround hacks I have in my code
I'm also open to other suggestions. I've been using SPIFFS because it's what I know, but would another file manager work better? I would suspect not, since the files would still have to be read from flash. Is there any other way of storing files that doesn't require flash operations? Preferably in a way that doesn't require manually specifying each file name, such as embedding files like I've seen in some examples. Being able to plop all my files into a folder and not think about it is a wonderful feature IMO. Or is it possible to disable the IPC feature somehow, or prevent SPIFFS from using it?
Debugging process
I had my LED code working great, and implemented a non-file-based web server. Everything still worked great, I didn't see any random color flashing at that point. As soon as I added file handling to the web server, I noticed the LEDs flashing random colors each time I loaded a web page. I commented out the file reading parts of my code, and the flashing went away. That was my first clue.
The WS2812 LEDs would only flash random colors if the timing sequence was incorrect, so I suspected the file reading was somehow interrupting the RMT driver. This is when I pinned the LED task to core 1 and the web server task to core 0, hoping that would prevent anything from interrupting the RMT module. But the LED still flashed each time a file was served. Even giving the LED task max priority didn't change anything.
I discovered vTaskList(), and found the LwIP task had no affinity. I pinned it to core 0, but it still didn't fix the problem. Now every tak is pinned to core 0, except for 3 of them: 1) my LED task, 2) an idle task (doesn't really do anything from what I can tell), and 3) the ipc1 task. (There's also the ipc0 task on core 0, but I assume that's not really relevant here)
I had no idea what the task was, so I tried Googling (it's really hard to search for 3 letter acronyms sometimes...). Those links I posted above are pretty much the only information I can find about the IPC, what it does, and how it's used. That one comment about the IPC tasks being used for flash was my other clue, so I thought the IPC task might be the culprit.
I managed to find the ipc.c file to see what it does, and it's not a lot (~150 lines of code including comments, which are admittedly sparse). I tried tinkering with it in a few ways (only creating 1 task, pinning both tasks to core 0, deleting the tasks, etc.), all of which severely broke everything
.
I did find a couple work arounds, sort of. The first is the one I've implemented in the snippet above, which seems to be fine. The other is to make FreeRTOS run only on the first core through the menuconfig. This prevents me from pinning my LED task to the other core, but it does prevent both IPC tasks from being created. Is there a way to use FreeRTOS on just core 0 while having my LED code run on core 1? Sounds like it could get tricky, but might be worth looking into if no other solution is found.
Conclusion
Any input on this would be greatly appreciated! I would love to know whether my workaround actually seems reasonable, I'm going to continue using it for now. If there are any other suggestions, I'd be happy to hear them! I might also be able to create a minimum example project that demonstrates the problem if someone would like that, though I don't have a lot of free time right now.
Again, should I create an issue in the IDF GitHub? Definitely seems like a bug to me, the LEDs shouldn't flash random colors just because I read a file!
I'm working on a project that implements a file-based web server to control a string of WS2812 LEDs. The web server by itself works great, and the LEDs by themselves work great. The problem is when I have both running simultaneously, the LEDs flash random colors any time a client gets a file from the web server. Here's some details about the LEDs and the web server:
LEDs - There's a chain of 114 WS2812 LEDs connected to pin 23, which are driven by the RMT module. There is a logic level converter before the first LED to convert the 3.3V signal to 5V for the LEDs. My code is fairly similar to the led_strip example in the IDF. I've created a dedicated LED task pinned to core 1 with max priority, everything else has been configured to run on core 0 (part of my debugging process, see debugging section below).
Web server - It's a normal HTTP server that sends files to HTTP clients using SPIFFS. Right now it's only sending a single text file with "Hello world" as its contents, so nothing intense. My code is fairly similar to the file_serving example in the IDF. This runs on core 0 along with everything else, except the LED task.
My diagnosis, workaround, and questions
After lots of digging through code (see debugging section below), I believe the problem has to do with the IPC feature interrupting the RMT driver. If I temporarily suspend the IPC task while the RMT module is sending data, there's no more flashing of random colors, everything works great. Here's a snippet of my code:
Code: Select all
vTaskSuspend(ipc1Handle);
rmt_write_items(RMT_CHANNEL_0, ledItems, numItems, true);
vTaskResume(ipc1Handle);
Suspending the IPC task like this seems to work well, but I suspect it's not recommended, and I haven't tested it thoroughly enough to feel confident in using it. There are very few config options for the IPC tasks, which leads me to believe that either 1) it's an essential thing that shouldn't be touched, or 2) it's something used so rarely that it's never been polished with more config options. Both sound plausible to me with my current knowledge of what the IPC tasks do (which is admittedly not much). Does anyone know?
Is this a bug? I feel like the RMT module shouldn't be interrupted under most circumstances, some things have very tight timing requirements (eg. WS2812 LEDs), and simply reading a file shouldn't interrupt that. Should I create an issue in the IDF GitHub? I would like to limit how many workaround hacks I have in my code
I'm also open to other suggestions. I've been using SPIFFS because it's what I know, but would another file manager work better? I would suspect not, since the files would still have to be read from flash. Is there any other way of storing files that doesn't require flash operations? Preferably in a way that doesn't require manually specifying each file name, such as embedding files like I've seen in some examples. Being able to plop all my files into a folder and not think about it is a wonderful feature IMO. Or is it possible to disable the IPC feature somehow, or prevent SPIFFS from using it?
Debugging process
I had my LED code working great, and implemented a non-file-based web server. Everything still worked great, I didn't see any random color flashing at that point. As soon as I added file handling to the web server, I noticed the LEDs flashing random colors each time I loaded a web page. I commented out the file reading parts of my code, and the flashing went away. That was my first clue.
The WS2812 LEDs would only flash random colors if the timing sequence was incorrect, so I suspected the file reading was somehow interrupting the RMT driver. This is when I pinned the LED task to core 1 and the web server task to core 0, hoping that would prevent anything from interrupting the RMT module. But the LED still flashed each time a file was served. Even giving the LED task max priority didn't change anything.
I discovered vTaskList(), and found the LwIP task had no affinity. I pinned it to core 0, but it still didn't fix the problem. Now every tak is pinned to core 0, except for 3 of them: 1) my LED task, 2) an idle task (doesn't really do anything from what I can tell), and 3) the ipc1 task. (There's also the ipc0 task on core 0, but I assume that's not really relevant here)
I had no idea what the task was, so I tried Googling (it's really hard to search for 3 letter acronyms sometimes...). Those links I posted above are pretty much the only information I can find about the IPC, what it does, and how it's used. That one comment about the IPC tasks being used for flash was my other clue, so I thought the IPC task might be the culprit.
I managed to find the ipc.c file to see what it does, and it's not a lot (~150 lines of code including comments, which are admittedly sparse). I tried tinkering with it in a few ways (only creating 1 task, pinning both tasks to core 0, deleting the tasks, etc.), all of which severely broke everything
.I did find a couple work arounds, sort of. The first is the one I've implemented in the snippet above, which seems to be fine. The other is to make FreeRTOS run only on the first core through the menuconfig. This prevents me from pinning my LED task to the other core, but it does prevent both IPC tasks from being created. Is there a way to use FreeRTOS on just core 0 while having my LED code run on core 1? Sounds like it could get tricky, but might be worth looking into if no other solution is found.
Conclusion
Any input on this would be greatly appreciated! I would love to know whether my workaround actually seems reasonable, I'm going to continue using it for now. If there are any other suggestions, I'd be happy to hear them! I might also be able to create a minimum example project that demonstrates the problem if someone would like that, though I don't have a lot of free time right now.
Again, should I create an issue in the IDF GitHub? Definitely seems like a bug to me, the LEDs shouldn't flash random colors just because I read a file!