Compatibility between "normal CMake" and ESP-IDF

[Oleg Endo]

Hi!

Despite the current crash problem with the ESP-IDF cmake build, and in the hope that it will be fixed at some point in time, I have been trying to integrate the cmake build with my own cmake based build setup. Actually my setup is similar to the ESP-IDF setup. It's interesting to see how you handle the linkerscripts. Of course there are several things which I wouldn't do the way you do it, but that's perhaps just my opinion.

One thing however, is a kind of showstopper. With your cmake build setup it's impossible to use any other cmake based projects, especially library projects, which I believe is one key aspect of using cmake in the first place. I've got a couple of library projects, coming from other cross platform developments, which I'd like to re-use on ESP32. The way you treat the concept of toolchain files and the override of the "project" macro/function makes everything fall apart unfortunately. The problem is that pretty much every cmake based project has a "project" usage in it and then you throw the whole ESP-IDF at it and treat it like an application project. Rewriting existing libraries to use your component approach does not seem feasible.

In the end, what I would like to do is something like this in the "main component" of an IDF based project:

Code: Select all

set (CMAKE_CXX_STANDARD 14)
set (CMAKE_CXX_STANDARD_REQUIRED 14)
set (CMAKE_CXX__EXTENSIONS OFF)

set(COMPONENT_SRCS "main.cpp")
set(COMPONENT_ADD_INCLUDEDIRS "")

register_component()

import_library (utils) // like "add_subdirectory" but with some enhancements
import_library (hash)  // like "add_subdirectory" but with some enhancements
target_link_libraries (main utils hash)

where "utils" and "hash" are some other existing library projects.

After some digging, it seems only a few changes are needed in your cmake setup to support this. Please see the attached patch. The key point is to rename your macro "project" to "idf_project" and pass in the toolchain file from OUTSIDE the build setup. This allows using different toolchain files when setting up the build configuration. As an example, I've also attached the toolchain file which I'm using. It's got some of those additions, which are pulled in by

Code: Select all

include (${CMAKE_CURRENT_LIST_DIR}/../common.cmake)

...but this is just an example use case.

The build is then setup for example like this

Code: Select all

cmake -DCMAKE_TOOLCHAIN_FILE=toolchain.cmake <esp idf project source dir>

With those things in place, I can use the ESP-IDF system as well as my own cmake system. For example, one of the libraries generates include header files (used by the cross compiler build) by running the host compiler to first build a host tool out of the same source code and things like that. It works normally as expected.

So hm yeah, this way your initial build setup step is a bit more complicated, because your users will need to specify the toolchain file separately when running cmake. But even cmake-gui has an option for that

I hope you can incorporate my suggestions into your system. It would make it much much more useful and easier to use with other existing cmake setups.

[permal]

I just began looking into using CMake for IDF, and I already find myself in a similar situation as Oleg describes above.

I compile and run large parts of my code on/for Linux using sanitizers etc. before even trying to run it on the ESP32. As such I've got regular CMake projects for my code already.

Although I think I see what you're trying to do by wrapping CMake-functionality to introduce the component-concept and some magic to detect code changes using git, I honestly don't think its the correct way to go. CMake may be somewhat tricky to get started with, but this really puts a spanner into the works. I't is my firm belief that a build system based on CMake must behave as per Kitware's documentation in all cases. CMake already supports components, they're just named "libraries" as per regular C/C++ nomenclature, why introduce something new for the same purpose?

I applaud added useful functions as much as anyone else, but I also expect to be able to use CMake according to the official documentation. Let us write libraries, then include those to our main app via add_subdirectory()/target_link_library() etc. Surely it is possible to keep CMake functionality intact while adding required build steps on top, without making regular use of CMake-functionality impossible?

Ideally, I'd like to see something like this in the main CMakeLists.txt file; regular CMAke commands with well isolated IDF-specific commands.

Code: Select all

cmake_minimum_required(3.6)
project(my_project)

include($ENV{IDF_PATH}/tools/cmake/idf_compile_tools.cmake)

set(SOURCES main.cpp main.h)
add_executable(${PROJECT_NAME} ${SOURCES})

idf_include_component(${PROJECT_NAME} cjson)
add_subdirectory(my_lib)
target_link_library(${PROJECT_NAME} my_lib cjson)

idf_do_whatever()

Preferably idf_* functions having to do with IDF-specific things would do nothing unless CMake is invoked with -DCOMPILE_FOR_IDF or similar. This is to allow using the same CMakeLists.txt for both native and ESP32 targets.

[ESP_Angus]

Hi Oleg & permal,

Thanks for the feedback, it's appreciated. You're certainly correct that we came to CMake with the intention of "can we port the build system we already have, in a way which seems familiar to existing ESP-IDF users?" rather than necessarily doing everything "the CMake way". This is the first time I've personally used CMake for non-trivial projects, so there's some learning curve here as well. And part of the reason it's been put up as a "preview" release and not yet the official build system is so that we can accommodate this kind of feedback.

To check I understand, there's two similar but different use cases being discussed here:

• Take a CMake library project with an existing CMakeList files and include it in an IDF application without needing to rewrite the CMakeLists file. At the non-project level, this is at least intended as a supported use case (it even has a short section in the docs) but I can see that for existing library projects it's going to fail. Oleg, do you have some examples of particular libraries you're trying this with - is any of the code publically available?
• Build an application from a single CMakeLists file in a way which lets it run on both host Linux and ESP32. This is a long term goal for ESP-IDF as well, although still some distance away. At the moment, the way that we expected this would work is that you build components as libraries in a cross platform way (see previous point) and have two top-level CMakeLists files, one for IDF and one for host builds (or one CMakeLists files with some if/else logic in them).

Regarding a few of the specifics:

After some digging, it seems only a few changes are needed in your cmake setup to support this. Please see the attached patch. The key point is to rename your macro "project" to "idf_project" and pass in the toolchain file from OUTSIDE the build setup. This allows using different toolchain files when setting up the build configuration.

It's funny, I think this is looking a lot like one of the early development versions of the CMake build. I originally wanted an idf_project macro, but stumbled across the problem where CMAKE_TOOLCHAIN_FILE had to be passed in externally or CMake would add an "implicit" call to project() in the top-level CMakeLists, and initialise with a host toolchain.

I understand why CMake is designed to have the toolchain file as a external cache override - it's a clean design choice for builds which are usually native and only occasionally cross built, usually on another "big" OS not for embedded. But it seems backwards in a cross-compile-oriented build system.

Long term, we're looking to support multiple toolchains and multiple chip-level targets in ESP-IDF, not just ESP32 + xtensa-esp32-elf-. So having "sensible defaults" for things like toolchains is desirable from the IDF side, even if it's heresy from the CMake side.

Will look into what we can do about idf_project(). The other possibility, that as it happens we're already looking into, is to have a separate .cmake include file (not project.cmake) that you can include if you want all the IDF support functions, but not the automatic "IDF magic" override of project(). This sounds like it would work for your use case as well, Oleg.

Let us write libraries, then include those to our main app via add_subdirectory()/target_link_library() etc. Surely it is possible to keep CMake functionality intact while adding required build steps on top, without making regular use of CMake-functionality impossible?

This was certainly intended as an existing use case, provided the "pure CMake" parts are added to the build after the call to project() in the IDF project - ie you do all the IDF-specific stuff, and then you end up with an executable target named "projectname.elf" that you can do whatever you like from there onwards.

Are you sure this is not possible now? (Problems with project() aside?)

CMake already supports components, they're just named "libraries" as per regular C/C++ nomenclature, why introduce something new for the same purpose?

The reason is: we have plans to expand components past their current level of functionality, to have an "IDF package manager" for installing components into a project (so the IDF "core" can slim down a bit, and to make it easier for users to share components with each other).

CMake also supports some package-manager-like features, and these were originally investigated as a way of getting this to all work in pure CMake. Unfortunately the CMake package features have a bunch of assumptions about "full-sized OS" host behaviour (from memory: installation, system libraries, that a library can build for any target, etc) built into them, so it didn't seem like they could be used as-is.

Ideally, I'd like to see something like this in the main CMakeLists.txt file; regular CMake commands with well isolated IDF-specific commands.

I think the sticking point here for "general" use cases is that with current IDF versions this hypothetical project can only work if it doesn't make any use of IDF features at all (ie only POSIX/libstdc++/etc features, no FreeRTOS and no drivers, etc.) This is why, long term, we'd like to support building with (a subset of) IDF components on the host OS.

However, we should be able to work towards a point where a CMakeLists file like this (probably not exactly like this, but close) can work for the use case you describe. (I feel like it could be made to work right now, but only with a lot of if/then type logic.)

[Oleg Endo]

Hi Angus,

Thanks for getting back to us!

ESP_Angus wrote:
Take a CMake library project with an existing CMakeList files and include it in an IDF application without needing to rewrite the CMakeLists file. At the non-project level, this is at least intended as a supported use case (it even has a short section in the docs) but I can see that for existing library projects it's going to fail. Oleg, do you have some examples of particular libraries you're trying this with - is any of the code publically available?

First, let's trade one JSON library with another
I'm using https://github.com/nlohmann/json
That gets pulled in as-is from github and then in my project I just do (adding it to my previous example above)

Code: Select all

set (JSON_BuildTests OFF CACHE BOOL "")
import_library (json)

target_link_libraries (main utils hash nlohmann_json)

And then things just magically work. That's the way it should be

Another example would be zlib. It also comes with its own CMakeLists.txt. Doesn't cooperate nicely out of the box unfortunately. Requires some patching in the CMakeLists.txt. I'm attaching my modified CMakeLists.txt for zlib. You can diff it against their release version to see my changes, it's not so much actually. I should submit my changes there one day. Anyway, with those, adding it to the app project is as straight forward as before:

Code: Select all

set (ZLIB_SHARED FALSE CACHE BOOL "")
import_library (zlib)

target_link_libraries (main utils hash nlohmann_json zlibstatic)

Another example is libpng. I'm using the public release version as-is and include it like:

Code: Select all

set (PNG_SHARED FALSE CACHE BOOL "")
set (PNG_TESTS FALSE CACHE BOOL "")
set (SKIP_INSTALL_ALL TRUE CACHE BOOL "")
set (AWK "" CACHE STRING "")
import_library (libpng)

target_link_libraries (main utils hash nlohmann_json zlibstatic libpngstatic)

libpng is a bit more advanced. It's got a dependency on libzlib and libm, which it will try to pull in like that in its CMakeLists.txt:

Code: Select all

  find_package(ZLIB REQUIRED)
  include_directories(${ZLIB_INCLUDE_DIR})
  ...
  find_library(M_LIBRARY
    NAMES m
    PATHS /usr/lib /usr/local/lib
  )

libzlib will be already built by the project and thus will be picked up by cmake naturally. libm however .... in my case, I'm using the "pre-installed" libm from newlib from the cross compiler toolchain build. The problem with all those "installed" libraries are, they are often multilibs, i.e. the actual library will depend on the compiler flags of the whole build. Currently cmake doesn't handle that at all and requires workarounds, but it can be made to work with some additions in the toolchain file.

I hope these can be useful examples for you.

ESP_Agnus wrote:
It's funny, I think this is looking a lot like one of the early development versions of the CMake build. I originally wanted an idf_project macro, but stumbled across the problem where CMAKE_TOOLCHAIN_FILE had to be passed in externally or CMake would add an "implicit" call to project() in the top-level CMakeLists, and initialise with a host toolchain.

I understand why CMake is designed to have the toolchain file as a external cache override - it's a clean design choice for builds which are usually native and only occasionally cross built, usually on another "big" OS not for embedded. But it seems backwards in a cross-compile-oriented build system.

We could argue what's backwards and what is forwards until the cows come home.
I think everybody building this kind of bigger system has a particular view on things. However, I do agree that cmake has some deficits when it comes to cross compiling, especially when it comes to compiling and running generator tools for the host system as part of the cross build. It totally lacks the notion of host toolchain vs. target toolchain. But it can be mitigated to some extent by using ExternalProject for example.
On top of that, most of the libraries out there are not written to be compiled as part of a bigger project -- cross compiling or not. Instead they are meant to be installed in the OS environment first, then used by other projects. That's one of the reasons why zlib requires some patching for instance. In our case (cross compiling everything from source for some MCU thingy) there is no such thing as "make install"... but that issue can be addressed. Modern examples (nlohmann_json) show that it can work nicely.

ESP_Agnus wrote:
The reason is: we have plans to expand components past their current level of functionality, to have an "IDF package manager" for installing components into a project (so the IDF "core" can slim down a bit, and to make it easier for users to share components with each other).

I totally understand your motivation and where this is heading. Although I'm not sure whether cmake is the right tool for the job. Like you said yourself, it's not really made for it. Maybe leave cmake being cmake and use something else for packet management like ebuild / emerge? Dunno ...

My view on this is, when I write an app, I want to decide which modules I want to use, at which versions and so on. I think git submodules and a top-level CMakeLists.txt for the executable project that manually pulls in all required libraries (either via standard add_subdirectory or some enhanced mechanism) is sufficient to get this done. For instance, in one MCU system, the ESP32 is an optional add-on and is also treated as such by the build. The CMakeLists.txt for the main application (another MCU, not ESP32) looks like that:

Code: Select all

cmake_minimum_required(VERSION 3.0)
include (ExternalProject)

project (wifi_demo)

ExternalProject_Add (esp32
  SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/esp32
  PREFIX esp32
  TMP_DIR esp32
  BINARY_DIR esp32
  CMAKE_ARGS -DCMAKE_TOOLCHAIN_FILE=${TOOLCHAIN_DIR}/../esp32-idf/toolchain.cmake -DCMAKE_LIBRARIES_DIR=${LIBRARIES_DIR}
  BUILD_ALWAYS 1
  BUILD_COMMAND $(MAKE)
  STEP_TARGETS build flash menuconfig
  EXCLUDE_FROM_ALL TRUE
)

ExternalProject_Add_Step (esp32 flash
  WORKING_DIRECTORY esp32
  COMMAND $(MAKE) flash
  DEPENDEES build
)

ExternalProject_Add_Step (esp32 menuconfig
  WORKING_DIRECTORY esp32
  COMMAND $(MAKE) menuconfig
)

add_executable (wifi_demo.esp32.bin IMPORTED)
set_property (TARGET wifi_demo.esp32.bin PROPERTY IMPORTED_LOCATION "${CMAKE_CURRENT_BINARY_DIR}/esp32/wifi_demo.bin")
add_dependencies (wifi_demo.esp32.bin esp32-build)

import_library (utils)
import_library (board)
import_library (uip)
import_library (net)
import_library (dev)
import_library (hash)
import_library (fs)
import_library (logging)
import_library (version_tag)

set (CMAKE_CXX_STANDARD 14)
set (CMAKE_CXX_STANDARD_REQUIRED 14)
set (CMAKE_CXX__EXTENSIONS OFF)

# enhanced version of the standard "add_executable"
add_target_executable (wifi_demo.elf main.cpp)

add_dependencies (wifi_demo.elf wifi_demo.esp32.bin)

target_link_libraries (wifi_demo.elf
  board board_crt stdc++ m c supc++ gcc
  uip
  net
  version_tag
  hash
  fs
  logging
  utils
  dev
)

# these are additional cmake functions provided by my "version_tag" library
set_default_version_tag_product_name ("WiFi Demo")
add_version_tag (wifi_demo.elf)

add_additional_executable_targets (wifi_demo.elf)

Surely, it's more build-code than the current IDF cmake files, but it also allows for more fine gained control over things, which is necessary for more advanced applications.

[permal]

Thanks for listening, Angus.

Are you sure this is not possible now? (Problems with project() aside?)

Well, project() stops you fairly early in the process, don't know if there are other things. I don't know why it happens, but I often got the error "add_definition" cannot be scripted while trying. I guess that is emitted when cmake is run with the -P flag. Can't reproduce right now though.

The reason is: we have plans to expand components past their current level of functionality, to have an "IDF package manager" for installing components into a project (so the IDF "core" can slim down a bit, and to make it easier for users to share components with each other).

I really think using CMake for that is the wrong way to go. I suggest simply setting a standard for how components should be packaged/distributed, perhaps with a python tool to help validate. "Installing" a component (for use with CMake's FindPackage()) might sound like a good idea, but please remember version management of such components. Git submodules solves that, including distribution.

[permal]

I just posted something closely related to what we're discussing in this thread - target management in CMake-enabled IDEs, such as CLion here.

As a non-IDF developer, there is normally little to no interest in the source of the IDF-components, except for the headers. Like I wrote in the linked post, its painful having to navigate around projects/targets that I have no interest in.

As such I'd really like to see this heading towards an end result where components are built separately from the main application as libraries and included via target_link_library() and its headers via include_directories(). If you actually are debugging an IDF component it can easily be added via add_subdirectory() by passing it a binary directory, like so:

Code: Select all

add_subdirectory("$ENV{IDF_PATH}/components/component" "${CMAKE_CURRENT_BINARY_DIR}/build_idf/component")

For IDF-devs, the solution is rather simple - a top level CMakeLists.txt in the IDF-tree that includes all the components as separate targets, just like currently happens.

Building IDF components as libraries has another advantage - it better decouples the components from the main application, which allows easier mocking/stubbing/substituting so that we can continue to develop and test our applications on the host OS without having to build completely separate projects for it. Being able to run sanitizers etc. on the portable part of the code base is worth a lot.

If you still want to provide a wrapper around the components, I think that'd work even if they are built separately from the main application via add_custom_target()/add_custom_command() that builds the component as a library, then an add_dependency(my_project idf_component_name)

Example

Code: Select all

project(my_project)
idf_add_required_components(my_project) # esptoop-py, bootloader, cxx etc.
idf_add_components(${PROJECT_NAME} json sdmmc wear_leveling)

[ESP_Angus]

As such I'd really like to see this heading towards an end result where components are built separately from the main application as libraries and included via target_link_library() and its headers via include_directories(). If you actually are debugging an IDF component it can easily be added via add_subdirectory() by passing it a binary directory, like so:

Code: Select all

add_subdirectory("$ENV{IDF_PATH}/components/component" "${CMAKE_CURRENT_BINARY_DIR}/build_idf/component")

The difficulty here is that we have "menuconfig", and configuration changes require components to recompile even if no source code changes.

But I see your point that currently the "signal to noise" for the average project in an IDE like CLion is very low. Will see what we can do about this.

[Oleg Endo]

permal wrote:
As such I'd really like to see this heading towards an end result where components are built separately from the main application as libraries and included via target_link_library() and its headers via include_directories().

From what I can see, they already are built as separate libraries. The problem is they are part of the top-level project. They should be separate projects instead.

ESP_Angus wrote:
The difficulty here is that we have "menuconfig", and configuration changes require components to recompile even if no source code changes.

I'd suggest to convert those to cmake cache variables and use ccmake or cmake-gui instead of menuconfig. But I know that this is a lot of hassle.

[permal]

From what I can see, they already are built as separate libraries. The problem is they are part of the top-level project. They should be separate projects instead.

Yeah, that was what I tried to say - build them via a separate process so that they are not part of the main application.

[permal]

Angus,

I've reached a point where I no longer can hold off the software side of the project. As such I'm going to start converting my framework into components and I just want to ask if there are any major changes coming to the CMake build system that, if known, can prevent double work.

As always, thanks for listening.