Multiple binaries per single ESP-IDF project?

[yaqwsx]

I am working on a project that is more-less a library. Therefore, I would like to have roughly the following project structure:

Code: Select all

- components
    - libraryComponent1
    - libraryComponent2
    - libraryComponent3
- examples
    - example1
    - example2
    - example3

Where examples are complete firmware for ESP32. I would like to make them easy to build for the users and especially, make them fast to compile and flash. When I make each example a separate ESP-IDF project (with root CMakeLists.txt), there is one problem - each example would have a separate build directory and ESP-IDF would be built for every example again and again. That is troublesome as it adds extra time to the compilation and also, every build directory is roughly 600MB, which adds up when you have more than 20 of them.

All of my examples share the exact same configuration, so I was wondering if it is possible to have or hack multiple binaries per one IDF project, so I reduce the usage of disk space and speed up the compilation. The goal is to allow users to compile all examples and then flash them one by one via a script (essentially giving the user an interactive show case).

[ESP_igrr]

Hi yaqwsx,

Unfortunately there isn't a way to do this while keeping the standard esp-idf project structure.

I think in most cases if sdkconfig files of your projects are the same, ccache should be able to reuse object files from the previous project build, speeding up compilation. Ccache is installed and enabled by default with ESP-IDF on Windows, and can be installed from OS package manager on Linux and Mac.

[yaqwsx]

Hi iggrr, thanks for the fast response!

I am aware of the cache solution. However, if I am not mistaken, it does not solve the problem of having multiple

I think it's too bad that the idf.py and whole CMake build system does not support multiple binaries per project. I think this absence is really painful. It would be really nice if we could have a CMake function to specify a binary in a project that is composed out of a list of components with a given SDK config (preferably an override of project-wise configuration). Then we could use commands like `idf.py build binaryA` or `idf.py build binaryB` or `idf.py flash binaryA`. It is a common practice for many projects to have multiple binaries. There is a number of use cases for that:

- when you write a library, you can have examples and your users can easily invoke `idf.py build all` to have them all ready to try.

- when you develop a multi-processor system where every processor needs a slightly different firmware (i.e., different `app_main`)

- it would allow us to write tests easily! Currently, writing on target test is extremely painful (the ESP-IDF documentation specifies you have to switch to the testing application somewhere in ESP-IDF, compile it with your components, meeh, so complicated!) Multiple binaries would allow us to have a target called `test` that would compile **in the project** the testing application for our components, flash it and run it. Also, I am getting into the situation when I have more tests that can reasonably fit inside the flash memory, so I have to split them into multiple binaries. In the current setup, it is overly complicated.

For me, the last point is the biggest added value - the possibility to seamlessly write & run both on-target and off-target tests. Could you please consider adding a proper way to compile and run tests on target (already discussed in IDFGH-54)?

I am wondering - as all reasonable build systems (including CMake) support multiple binaries/targets, why did you decide to opt-out when building the whole ESP-IDF overlay layer?

PS: I think the whole multi-target situation could work well even for various skconfigs - the only thing that would need to change is that the components do not register the build process directly, instead, they define a function that registers the build process for given sdk config. In that ways, multiple sdk configs could easily co-exist.

[ESP_igrr]

Hi yaqwsx,

Thanks for explaining the use cases. I agree that the ability to build multiple binaries in a single project would be useful. Unfortunately we didn't consider this as one of the requirements when ESP-IDF build system was developed, which is the reason why it's not supported.

I looked at this issue about a year ago, and I recall that building multiple .elf files was actually possible. What didn't work was generating multiple binaries, since esptool_py component uses generator expressions to create the binary targets. Here's the basic idea how to build multiple ELF files (note, i took this from the internal bug report i opened a while ago; i haven't verified if this still works with recent IDF versions).

Code: Select all

project(multi_app include($ENV{IDF_PATH}/tools/cmake/idf.cmake)

idf_build_component(main_one)
idf_build_component(main_two)

idf_build_process(esp32
                  SDKCONFIG ${CMAKE_BINARY_DIR}/sdkconfig
                  BUILD_DIR ${CMAKE_BINARY_DIR})

add_executable(one.elf dummy.c)
target_link_libraries(one.elf idf::main_one)
idf_build_executable(one.elf)

add_executable(two.elf dummy.c)
target_link_libraries(two.elf idf::main_two)
idf_build_executable(two.elf)

I didn't understand your point in the "PS" regarding support for multiple sdkconfigs, though. The main issue with supporting multiple sdkconfigs is that each component build depends on sdkconfig. If we need to support multiple sdkconfigs in a single build, that means that for each pair of {sdkconfig, component} we need to create a separate build system target. This is possible, but would require very extensive changes to the build system. For example, currently knowing a component name is assumed to be sufficient to find the library built from this component. If multiple sdkconfigs are supported in a single build, this is no longer the case, and we need to specify sdkconfig as well as the component name to find the correct library. I'm afraid this will require very significant changes to be viable...

[ESP_igrr]

I took a look at this again, here's the current status:

• Building multiple ELF files is possible using a snippet similar to the one posted above.
• Only the last ELF file is currently considered for the purpose of binary image generation. In the example above, we would get "two.bin" built but not "one.bin". Running "idf.py flash" would flash "two.bin", and "idf.py monitor" would assume that "two.elf" is running on the device.
• Adapting the build system to build more than one binary seems feasible. However the documented build properties "EXECUTABLE_NAME" and "EXECUTABLE" might need to be converted into lists, which is potentially a breaking change. There is also a problem that these properties need to be used in CMake generator expressions, and I'm not sure how to iterate over these lists inside a generator expression to build multiple binaries. There are also few other things that need to be adjusted, for example secure boot binary signing. If all this is done, then building multiple binaries with CMake (directly, not via idf.py) will work.
• The build system produces several metadata files which are used by idf.py and the IDE plugins (e.g. VS Code). Of interest here are project_description.json and flasher_args.json. Both file formats haven't been designed to accommodate more than one binary and ELF file. The formats can be changed, but this would be a breaking change for the IDE plugins and other software which currently uses these files.
• idf.py flash will need to get a new argument to decide which application to flash — this is simple enough if the flasher_args.json is updated to include the information about all binaries.
• "idf.py monitor" will need to know, somehow, which binary has previously been flashed into the device, to choose the corresponding .elf file. This is required for backtrace and core dump decoding. I can't think of any other solution other than running `idf.py monitor <app name>` and relying on the user to specifying the same name for "idf.py flash" and "idf.py monitor". There is also a usability issue that the name would have to be specified twice ("idf.py flash appname monitor appname") so that each command knows the app name. Alternatively, the app name could be added as a global argument ("idf.py --app appname flash monitor").
• IDE plugins will need to support discovering the available applications and provide some UI to choose which one to flash/monitor.

I think the changes required to project_description.json and flasher_args.json are the biggest concern, since we need to avoid breaking changes for other projects which rely on these files.