ESP32 Forum

ESP32 Official Forum
http://forum.esp32.com/

Executing merge_bin using Gitlab CI/CD

http://forum.esp32.com/viewtopic.php?f=2&t=37297

Page 1 of 1

Executing merge_bin using Gitlab CI/CD

Posted: Wed Dec 13, 2023 11:28 am
by zazas321
Hello. I am learning about CI/CD and how can I optimize my development.

I am running a docker on my VM.

Normally, whenever we do a new tag in Gitlab we want to also do a release. I have solved this by using the following .gitlab-ci.yml config:

Code: Select all

stages: # List of stages for jobs, and their order of execution
    - release

release_job:
    stage: release
    image: registry.gitlab.com/gitlab-org/release-cli:latest
    rules:
        - if: $CI_COMMIT_TAG # Run this job when a tag is created
    script:
        - echo "running release_job"
    release: # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
        tag_name: "$CI_COMMIT_TAG"
        description: "$CI_COMMIT_TAG"

I have also recently found out that esp-idf has its own docker image so I thought it would make sense to also do some optimizations.
I want to be able to execute the following command when we create a new Gitlab tag.

Code: Select all

esptool.py --chip esp32 merge_bin -o flash_image.bin @flash_args

So I have updated my .gitlab-ci.yml:

Code: Select all

stages: # List of stages for jobs, and their order of execution
    - release
    - build

release_job:
    stage: release
    image: registry.gitlab.com/gitlab-org/release-cli:latest
    rules:
        - if: $CI_COMMIT_TAG # Run this job when a tag is created
    script:
        - echo "running release_job"
    release: # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
        tag_name: "$CI_COMMIT_TAG"
        description: "$CI_COMMIT_TAG"


build-esp-idf-5.1:
  image: espressif/idf:release-v5.1
  variables:
    GIT_SUBMODULE_STRATEGY: normal
  stage: build
  rules:
    - if: $CI_COMMIT_TAG # Run this job when a tag is created
  script:
    - cd build; esptool.py --chip esp32 merge_bin -o flash_image.bin @flash_args
After running a job, I got the following error:

Code: Select all

Running with gitlab-runner 16.6.1 (f5da3c5a)
  on VM Docker runner MxeEWKAQ, system ID: s_1de69ef528ed
Preparing the "docker" executor
00:09
Using Docker executor with image espressif/idf:release-v5.1 ...
Pulling docker image espressif/idf:release-v5.1 ...
Using docker image sha256:c0408e2a8c84b96140d4003d8b427652a407577b4c159f444c27d59df447a6ad for espressif/idf:release-v5.1 with digest espressif/idf@sha256:ce81ec99a20f76fb0991a7f5e4824d40752542a5e0d2aaa3c353113890120be0 ...
Preparing environment
00:01
Running on runner-mxeewkaq-project-120-concurrent-0 via ems-gitlab...
Getting source from Git repository
00:02
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/ci_cd/esp32_application/.git/
Checking out 9e980409 as detached HEAD (ref is v2.2.0)...
Updating/initializing submodules with git depth set to 20...
Updated submodules
Executing "step_script" stage of the job script
00:09
Using docker image sha256:c0408e2a8c84b96140d4003d8b427652a407577b4c159f444c27d59df447a6ad for espressif/idf:release-v5.1 with digest espressif/idf@sha256:ce81ec99a20f76fb0991a7f5e4824d40752542a5e0d2aaa3c353113890120be0 ...
Detecting the Python interpreter
Checking "python3" ...
Python 3.8.10
"python3" has been detected
Checking Python compatibility
Checking other ESP-IDF version.
Adding ESP-IDF tools to PATH...
Checking if Python packages are up to date...
Python requirements are satisfied.
Requirement files:
 - /opt/esp/idf/tools/requirements/requirements.core.txt
Python being checked: /opt/esp/python_env/idf5.1_py3.8_env/bin/python
Added the following directories to PATH:
  /opt/esp/idf/components/espcoredump
  /opt/esp/idf/components/partition_table
  /opt/esp/idf/components/app_update
  /opt/esp/tools/xtensa-esp-elf-gdb/12.1_20221002/xtensa-esp-elf-gdb/bin
  /opt/esp/tools/riscv32-esp-elf-gdb/12.1_20221002/riscv32-esp-elf-gdb/bin
  /opt/esp/tools/xtensa-esp32-elf/esp-12.2.0_20230208/xtensa-esp32-elf/bin
  /opt/esp/tools/xtensa-esp32s2-elf/esp-12.2.0_20230208/xtensa-esp32s2-elf/bin
  /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin
  /opt/esp/tools/riscv32-esp-elf/esp-12.2.0_20230208/riscv32-esp-elf/bin
  /opt/esp/tools/esp32ulp-elf/2.35_20220830/esp32ulp-elf/bin
  /opt/esp/tools/cmake/3.24.0/bin
  /opt/esp/tools/openocd-esp32/v0.12.0-esp32-20230921/openocd-esp32/bin
  /opt/esp/python_env/idf5.1_py3.8_env/bin
  /opt/esp/idf/tools
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:
  idf.py build
$ echo before script
before script
$ cd build; esptool.py --chip esp32 merge_bin -o flash_image.bin @flash_args
/usr/bin/bash: line 151: cd: build: No such file or directory
Running after_script
00:08
Detecting the Python interpreter
Checking "python3" ...
Python 3.8.10
"python3" has been detected
Checking Python compatibility
Checking other ESP-IDF version.
Adding ESP-IDF tools to PATH...
Checking if Python packages are up to date...
Python requirements are satisfied.
Requirement files:
 - /opt/esp/idf/tools/requirements/requirements.core.txt
Python being checked: /opt/esp/python_env/idf5.1_py3.8_env/bin/python
Added the following directories to PATH:
  /opt/esp/idf/components/espcoredump
  /opt/esp/idf/components/partition_table
  /opt/esp/idf/components/app_update
  /opt/esp/tools/xtensa-esp-elf-gdb/12.1_20221002/xtensa-esp-elf-gdb/bin
  /opt/esp/tools/riscv32-esp-elf-gdb/12.1_20221002/riscv32-esp-elf-gdb/bin
  /opt/esp/tools/xtensa-esp32-elf/esp-12.2.0_20230208/xtensa-esp32-elf/bin
  /opt/esp/tools/xtensa-esp32s2-elf/esp-12.2.0_20230208/xtensa-esp32s2-elf/bin
  /opt/esp/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin
  /opt/esp/tools/riscv32-esp-elf/esp-12.2.0_20230208/riscv32-esp-elf/bin
  /opt/esp/tools/esp32ulp-elf/2.35_20220830/esp32ulp-elf/bin
  /opt/esp/tools/cmake/3.24.0/bin
  /opt/esp/tools/openocd-esp32/v0.12.0-esp32-20230921/openocd-esp32/bin
  /opt/esp/python_env/idf5.1_py3.8_env/bin
  /opt/esp/idf/tools
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:
  idf.py build
Running after script...
$ echo after script
after script
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1
It failed to execute command because it cannot access the build folder of my repository. I want to understand a little bit more about how can I access the project build directory and merge all binaries via docker. After the binaries are merged, I would like to understand how can I access them? How can I configure the docker to provide me with the merged binary?

I would appreciate any help regarding setting up CI/CD pipeline for docker executor that would be capable of doing this

Re: Executing merge_bin using Gitlab CI/CD

Posted: Wed Dec 13, 2023 12:51 pm
by limpens
Not using gitlab here, but forgejo, so perhaps my syntax differs from yours, but, when using the esp-idf image, you still have to call

Code: Select all

idf.py build
don't you?

Maybe it is as simple as adding

Code: Select all

idf.py build;
before cd build command?
In my use case, I need to source the export.sh and can issue an idf.py build. And if everything is ok, I use curl to upload the result to a place for OTA actions.

Re: Executing merge_bin using Gitlab CI/CD

Posted: Thu Dec 14, 2023 9:27 am
by zazas321
Hey limpens. Thanks for your response. You were right. I was missing idf.py build. My .gitlab-ci.yml file looks like:

Code: Select all

stages: # List of stages for jobs, and their order of execution
    - build
    - release


build-esp-idf-5.1:
  image: espressif/idf:release-v5.1
  variables:
    GIT_SUBMODULE_STRATEGY: normal
  stage: build
  before_script:
    - idf.py build
  script:
    - mkdir full_binary; cd build; esptool.py --chip esp32 merge_bin -o ../full_binary/full_image.bin @flash_args


release_job:
    stage: release
    image: registry.gitlab.com/gitlab-org/release-cli:latest
    rules:
        - if: $CI_COMMIT_TAG # Run this job when a tag is created
    script:
        - echo "running release_job"
    release: # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
        tag_name: "$CI_COMMIT_TAG"
        description: "$CI_COMMIT_TAG"


I can now build my project and generate merged binary.
However, there is one more thing that is not fully clear to me.

Ideally, after generating merged binary, I would like to add this file to the release assets. For example, when the below job is executed:

Code: Select all

release_job:
    stage: release
    image: registry.gitlab.com/gitlab-org/release-cli:latest
    rules:
        - if: $CI_COMMIT_TAG # Run this job when a tag is created
    script:
        - echo "running release_job"
    release: # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
        tag_name: "$CI_COMMIT_TAG"
        description: "$CI_COMMIT_TAG"
A release is created that contains source files. Those source files themselves do not contain build folder or merged binary. Is there a way to link these 2 seperate jobs together (One is building and generating merged binary and the other is creating a release)?

I would like to add merged_binary to the release assets as shown in the image below:
https://ibb.co/KmFH7mt


You mentioned that you are not using Gitlab so you will most likely not going to be familiar with that.

Re: Executing merge_bin using Gitlab CI/CD

Posted: Fri Dec 15, 2023 5:31 am
by zazas321
I have managed to achieve this. See the full CI script:

Code: Select all

stages: # List of stages for jobs, and their order of execution
  - build
  - release


build-esp-idf-5.0.4:
  image: espressif/idf:v5.0.4
  variables:
    GIT_SUBMODULE_STRATEGY: normal

  stage: build
  rules:
    - if: $CI_COMMIT_TAG # Run this job when a tag is created
  before_script:
    - idf.py build
  script:
    - mkdir full_binary; 
    - cd build; 
    - esptool.py --chip esp32 merge_bin -o ../full_binary/${CI_PROJECT_NAME}_${CI_COMMIT_TAG}_full.bin @flash_args
    - cd ..
  after_script:
    - ls -l
    - cd full_binary
    - ls -l
    - touch JOB_ID.txt
    - echo GE_JOB_ID=$CI_JOB_ID >> JOB_ID.txt
    - cat JOB_ID.txt
    - ls -l
  artifacts:
    paths:
      - full_binary/${CI_PROJECT_NAME}_${CI_COMMIT_TAG}_full.bin
    reports:
      # To ensure we've access to this file in the next stage
      dotenv: full_binary/JOB_ID.txt



release_job:
    stage: release
    image: registry.gitlab.com/gitlab-org/release-cli:latest
    needs:
      - job: build-esp-idf-5.0.4
        artifacts: true
    rules:
      - if: $CI_COMMIT_TAG # Run this job when a tag is created
    script:
      - echo "running release_job"
      - echo $CI_PROJECT_URL
    release: # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
      tag_name: "$CI_COMMIT_TAG"
      description: "$CI_COMMIT_TAG"
      assets:
        links:
          - name: '${CI_PROJECT_NAME}_${CI_COMMIT_TAG}_full'
            url: '$CI_PROJECT_URL/-/jobs/${GE_JOB_ID}/artifacts/file/full_binary/${CI_PROJECT_NAME}_${CI_COMMIT_TAG}_full.bin'

The only prerequisites to get this working:

Creating gitlab runner with docker executor
All times are UTC
Page 1 of 1
Powered by phpBB® Forum Software © phpBB Limited
https://www.phpbb.com/