clarification about unit testing

[atlascoder]

Hello, everyone!

Sorry in advance for likely stupid questions but..
I am trying to write some general tests for my C code, but I don't clearly understand how to create unit-tests and perform testing in the project structure.

So, I have a project, which is located in ${IDF_PATH}/my_proj.
Also I have my custom component schedule in path ${IDF_PATH}/my_proj/components/schedule.
Following the Unit Testing in ESP32 chapter, I have created

${IDF_PATH}/my_proj/components/schedule/test
- test_schedule.c
- component.mk


Now, I need to write my test code, but it requires to include unity.h which is a component of the unit-test-app project, and I am confused with this.

So, what is the right way:
a) to copy (or symlink?) my schedule folder from my_proj/components to unit-test-app/components
b) to move my schedule component under ${IDF_PATH}/components
c) to move unit-test-app/components/unity to my_proj/components/unity
?

And another question. What approach could you recommend to perform testing on building host? After all, it's not always needed to test some general stuff on ESP and it's much faster to run such tests on building host.

[ESP_Angus]

Now, I need to write my test code, but it requires to include unity.h which is a component of the unit-test-app project, and I am confused with this.

Unit tests always run in the context of the unit-test-app project (ie not your project.)

To make it see your "schedule" directory, you should be able to do something like this:

Code: Select all

make -C ${IDF_PATH}/tools/unit-test-app EXTRA_COMPONENT_DIRS=/path/to/my_proj/components TEST_COMPONENTS=schedule

This should build the unit test app, but have it also look in your project's "components" directory for the "schedule" component's unit tests.

You can probably wrap this command into some custom "make unit-test" and "make unit-test-flash" targets in your project's own Makefile.

If this seems a little fiddly, it's because the original use case was testing IDF's internal components. If you have any ideas/suggestions about how we can make it more streamlined for use cases like yours, we'd love to hear them.

And another question. What approach could you recommend to perform testing on building host? After all, it's not always needed to test some general stuff on ESP and it's much faster to run such tests on building host.

This is a good approach (testing as much of your firmware logic on the host as possible). IDF itself doesn't provide much direct support for this. However some components have "host-side-test" support, look at the nvs_flash, mdns, heap and wear_levelling components for some ideas.

Exactly how to structure such tests probably depends on what your firmware does, and what layer you want to test at.

[atlascoder]

Thanks for this informative answer, ESP_Angus!

I think, the code with EXTRA_COMPONENT_DIRS is much suit for my case!

Code: Select all

make -C ${IDF_PATH}/tools/unit-test-app EXTRA_COMPONENT_DIRS=/path/to/my_proj/components TEST_COMPONENTS=schedule

And I will try to make some make target to build unit-test-app with my project's components.

As an idea about testing, just one thing has confused me - it's usage of the separate project for testing. If the testing project could be generated within an app project structure - it would be more convenient and coherent, I think. In example, I use template for a project as usual, an if I need to create some tests - I just run some script, or make target, that generates/includes necessary components and code in my project's structure along with it. And I get targets for building unit-test-app from my project's structure.

Yes, I understand that the most relevant tests should be executed in production-clone context. I don't know if used testing frameworks are able to run tests on building host, but it's more sophisticated thing because it may require to implement stubs/mocks for contextual things, and understood, it can't be easy to get.

[ESP_Angus]

As an idea about testing, just one thing has confused me - it's usage of the separate project for testing. If the testing project could be generated within an app project structure - it would be more convenient and coherent, I think. In example, I use template for a project as usual, an if I need to create some tests - I just run some script, or make target, that generates/includes necessary components and code in my project's structure along with it. And I get targets for building unit-test-app from my project's structure.

I understand.

For what it's worth, you can make your own unit test project that uses the IDF "unity" component by setting EXTRA_COMPONENT_DIRS to include tools/unit-test-app/components. But you'll need to write the "main" routine of the test app yourself, then.

Yes, I understand that the most relevant tests should be executed in production-clone context. I don't know if used testing frameworks are able to run tests on building host, but it's more sophisticated thing because it may require to implement stubs/mocks for contextual things, and understood, it can't be easy to get.

Unity can be used on the host (it's a platform-agnostic C library, see https://www.throwtheswitch.org/unity ). But you'd probably need a separate copy of it that gets included when building host-side tests. And, as you say, you need the additional mock hardware layers that are relevant to whatever you want to test.

[atlascoder]

Ok, I've got it worked.

I just added reference to unit-test-app/components/unity to my project's includes - it allowed to me edit tests' code within my project in Eclipse.

Then i added a phony target for building unit-test-app project with EXTRA_COMPONENTS_DIR, as you, Angus, suggested:

Code: Select all

make -C ${IDF_PATH}/tools/unit-test-app EXTRA_COMPONENT_DIRS=${IDF_PATH}/my_project/components TEST_COMPONENTS='schedule'

And then I also create another make target with command for flashing unit-test-app.

Now, I can easy add and edit my test code and run my app or the unit-test-app conveniently. Thanks for help!
***

And another question is mocking.. I dived deeply and met projects Ceedling, CMock and Fake Function Framework.

Ceedling http://www.throwtheswitch.org/ceedling/ allows to run tests on a development host. It generates directories and staff for new one project or for an existing one. I didn't try, because I found that it uses `build` directory and it would clash with existing one (http://www.electronvector.com/blog/add- ... h-ceedling), and I would not like to rebuild everything when switching projects. But this is very matching to idea expressed by me here about generating code.

CMock - is a mocking feature using the Unity. It has specific requirements to directory structure, which is different from esp-idf, particularly, all code files are under `src` dirs. But I guess it could be adopted to use with the unit-test-app project.

And Fake Function Framework (FFF) is simplest feature requiring for just one header file inclusion!

Unfortunately, I am not too proficient it building systems and all these mocking stuff requires for neat tuning of building scripts, as far I see. I've tried FFF and stuck into build error `multiple definition if xxx`. I know that to resolve it, I need to exclude mocked function from build (and its components as whole), but it's not seems easy yet.

ESP_Angus, probably you have some advice for further steps in using of mocks?

[atlascoder]

Howdy, again!

And yet, I'd like to return to mocking.. I am trying to build only certain components from ESP-IDF and my project in order to represent excluded components with fake functions.

I use the command:

Code: Select all

make -C ${IDF_PATH}/tools/unit-test-app EXTRA_COMPONENT_DIRS=${IDF_PATH}/my_proj/components TEST_COMPONENTS="workcycle" COMPONENTS="freertos esp32 newlib esptool_py nvs_flash spi_flash log tcpip_adapter lwip main xtensa-debug-module driver bt soc unity workcycle"

But I have following error:

Code: Select all

esp-idf/components/freertos/include/freertos/FreeRTOSConfig.h:73:23: fatal error: sdkconfig.h: No such file or directory

The sdkconfig.h present under build/include in both: my_project and unit-test-app project. I have tried to add -I build/include to the make arguments. but no luck.

How could I build unit-test-app with selected set of components, or probably, exclude a set of components?

[detlier]

Just want to add: I also found this very confusing. The documentation starts off by showing how to add your own unit tests in a component, but it turns out it's only meant for ESP-IDF's internal components, so who exactly is the documentation targeting?

That aside, how could I achieve this via CMake?

[ESP_igrr]

We have recently added example of creating unit tests and unit test projects: https://github.com/espressif/esp-idf/bl ... /README.md

[alejo9118]

I wrote an example showing how FFF (Fake Function Framework) can be used with the Unity component of ESP-IDF to write unit tests for your application with test doubles (mocks).
https://gitlab.com/deltalejo/esp32-unity-fff-demo