How to use alternate esptool in esp-idf-v446?

[jdupre]

I am using idf v4.4.6 in ubuntu 22.04.03 running in a Windows VirtualBox host.
I am unable to flash over serial cable with the version of esptool that is included with v4.4.6. However, the current version of esptool on github works fine. I thought I could just edit this section of export.sh to substitute the esptool in components/esptool_py/esptool/ with a different version:

Code: Select all

# Allow calling some IDF python tools without specifying the full path
# ${IDF_PATH}/tools is already added by 'idf_tools.py export'
# Comment out: IDF_ADD_PATHS_EXTRAS="${IDF_PATH}/components/esptool_py/esptool"
# Use this version of esptool insted: 
IDF_ADD_PATHS_EXTRAS="${HOME}/repos/esptool"
IDF_ADD_PATHS_EXTRAS="${IDF_ADD_PATHS_EXTRAS}:${IDF_PATH}/components/espcoredump"
IDF_ADD_PATHS_EXTRAS="${IDF_ADD_PATHS_EXTRAS}:${IDF_PATH}/components/partition_table"
IDF_ADD_PATHS_EXTRAS="${IDF_ADD_PATHS_EXTRAS}:${IDF_PATH}/components/app_update"
export PATH="${IDF_ADD_PATHS_EXTRAS}:${PATH}"

However, this does not work. It appears the path to "${IDF_PATH}/components/esptool_py/esptool" is hard-coded somewhere. But Where? (For one, it is hard-coded in idf.py on line 502, but that is just an informational message.)

I've traced it down to where esp-idf-v446\tools\idf_py_actions\tools.py calls "ninja flash" as a subprocess.

This is where I am lost. I know nothing about how ninja works. Where is ninja finding what esptool to use if not from the path?

Code: Select all

Executing action: flash

task: <__main__.init_cli.<locals>.Task object at 0x7fc5b8097df0>

ctx: <click.core.Context object at 0x7fc5b9540940>

global_args: {'verbose': None, 'port': '/dev/ttyUSB0', 'project_dir': '/home/jmd/repos/wcb', 'build_dir': '/home/jmd/repos/wcb/build', 'cmake_warn_uninitialized': False, 'preview': False, 'ccache': False, 'generator': None, 'dry_run': False, 'define_cache_entry': [], 'baud': 460800}

task.action_args: {}

Running ninja in directory /home/jmd/repos/wcb/build

Executing "ninja flash"...

jmd: Is this it?

jmd: args: ['ninja', 'flash']

jmd: env_copy: {'SHELL': '/bin/bash', 'SESSION_MANAGER': 'local/jmd-VirtualBox:@/tmp/.ICE-unix/2763,unix/jmd-VirtualBox:/tmp/.ICE-unix/2763', 'QT_ACCESSIBILITY': '1', 'COLORTERM': 'truecolor', 'XDG_CONFIG_DIRS': '/etc/xdg/xdg-ubuntu:/etc/xdg', 'SSH_AGENT_LAUNCHER': 'gnome-keyring', 'XDG_MENU_PREFIX': 'gnome-', 'GNOME_DESKTOP_SESSION_ID': 'this-is-deprecated', 'GNOME_SHELL_SESSION_MODE': 'ubuntu', 'SSH_AUTH_SOCK': '/run/user/1000/keyring/ssh', 'XMODIFIERS': '@im=ibus', 'DESKTOP_SESSION': 'ubuntu', 'IDF_PYTHON_ENV_PATH': '/home/jmd/.espressif/python_env/idf4.4_py3.10_env', 'GTK_MODULES': 'gail:atk-bridge', 'PWD': '/home/jmd/repos/wcb', 'LOGNAME': 'jmd', 'XDG_SESSION_DESKTOP': 'ubuntu', 'XDG_SESSION_TYPE': 'wayland', 'SYSTEMD_EXEC_PID': '2796', 'IDF_PATH': '/home/jmd/esp/esp-idf-v446', 'XAUTHORITY': '/run/user/1000/.mutter-Xwaylandauth.FX4DH2', 'OPENOCD_SCRIPTS': '/home/jmd/.espressif/tools/openocd-esp32/v0.12.0-esp32-20230419/openocd-esp32/share/openocd/scripts', 'HOME': '/home/jmd', 'USERNAME': 'jmd', 'IM_CONFIG_PHASE': '1', 'LANG': 'en_US.UTF-8', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'XDG_CURRENT_DESKTOP': 'ubuntu:GNOME', 'VTE_VERSION': '6800', 'WAYLAND_DISPLAY': 'wayland-0', 'GHPAT': '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', 'GNOME_TERMINAL_SCREEN': '/org/gnome/Terminal/screen/84ea3a0c_cbf9_4b9b_b581_e342f80bfde9', 'GNOME_SETUP_DISPLAY': ':1', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'XDG_SESSION_CLASS': 'user', 'TERM': 'xterm-256color', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'USER': 'jmd', 'GNOME_TERMINAL_SERVICE': ':1.112', 'DISPLAY': ':0', 'SHLVL': '1', 'QT_IM_MODULE': 'ibus', 'XDG_RUNTIME_DIR': '/run/user/1000', 'XDG_DATA_DIRS': '/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop', 'PATH': '/home/jmd/repos/esptool:/home/jmd/esp/esp-idf-v446/components/espcoredump:/home/jmd/esp/esp-idf-v446/components/partition_table:/home/jmd/esp/esp-idf-v446/components/app_update:/home/jmd/.espressif/tools/xtensa-esp-elf-gdb/11.2_20220823/xtensa-esp-elf-gdb/bin:/home/jmd/.espressif/tools/xtensa-esp32-elf/esp-2021r2-patch5-8.4.0/xtensa-esp32-elf/bin:/home/jmd/.espressif/tools/esp32ulp-elf/2.35_20220830/esp32ulp-elf/bin:/home/jmd/.espressif/tools/openocd-esp32/v0.12.0-esp32-20230419/openocd-esp32/bin:/home/jmd/.espressif/python_env/idf4.4_py3.10_env/bin:/home/jmd/esp/esp-idf-v446/tools:/home/jmd/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin', 'IDF_TOOLS_EXPORT_CMD': '/home/jmd/esp/esp-idf-v446/export.sh', 'IDF_TOOLS_INSTALL_CMD': '/home/jmd/esp/esp-idf-v446/install.sh', 'GDMSESSION': 'ubuntu', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'OLDPWD': '/home/jmd/repos', '_': '/home/jmd/esp/esp-idf-v446/tools/idf.py', 'PYTHON': '/home/jmd/.espressif/python_env/idf4.4_py3.10_env/bin/python', 'ESPBAUD': '460800', 'ESPPORT': '/dev/ttyUSB0'}

jmd: cwd /home/jmd/repos/wcb/build

jmd: running subprocess now...

[1/6] cd /home/jmd/repos/wcb/build/esp....bin /home/jmd/repos/wcb/build/wcb.bin

wcb.bin binary size 0x1751b0 bytes. Smallest app partition is 0x190000 bytes. 0x1ae50 bytes (7%) free.

[2/6] Performing build step for 'bootloader'

[1/1] cd /home/jmd/repos/wcb/build/bootloader/esp-idf/esptool_py && /home/jmd/.espressif/python_env/idf4.4_py3.10_env/bin/python /home/jmd/esp/esp-idf-v446/components/partition_table/check_sizes.py --offset 0xc000 bootloader 0x1000 /home/jmd/repos/wcb/build/bootloader/bootloader.bin

Bootloader binary size 0x6a30 bytes. 0x45d0 bytes (40%) free.

[3/4] cd /home/jmd/esp/esp-idf-v446/co...nents/esptool_py/run_serial_tool.cmake

esptool.py esp32 -p /dev/ttyUSB0 -b 460800 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 40m --flash_size 16MB 0x1000 bootloader/bootloader.bin 0x30000 wcb.bin 0xc000 partition_table/partition-table.bin 0x2b000 ota_data_initial.bin 0x3fc000 storage.bin

esptool.py v3.3.4-dev

Serial port /dev/ttyUSB0

Connecting......................................



A fatal error occurred: Failed to connect to ESP32: Wrong boot mode detected (0x17)! The chip needs to be in download mode.

For troubleshooting steps visit: https://docs.espressif.com/projects/esptool/en/latest/troubleshooting.html

CMake Error at run_serial_tool.cmake:56 (message):

  /home/jmd/.espressif/python_env/idf4.4_py3.10_env/bin/python

  /home/jmd/esp/esp-idf-v446/components/esptool_py/esptool/esptool.py --chip

  esp32 failed

I have come up with a few hacks to fix this, but shouldn't setting the path to esptool.py "just work"?

--Joe

[ESP_Radim]

Hello,

I have come up with a few hacks to fix this, but shouldn't setting the path to esptool.py "just work"?

This is not the full path to the tool used by the build system. It is just a convenience (to Allow calling some IDF python tools without specifying the full path).

ESP-IDF v4.4 is only fully compatible with esptool v3.*. Using a more recent version of esptool (v4.*) is not guaranteed to work and break anything in the ESP-IDF environment.

Using a separate esptool instance:
If you want, you can install the latest version of esptool into a separate environment and use it directly. The instructions on how to do that can be found in the esptool documentation. You can then run the build in the IDF environment and flash the app in a separate environment by copying and running the flash command generated by the build system (printed after the "Project build complete. To flash, run this command:" message). Just make sure you are in the new separate environment and you changed the command to work in it (e.g. changing the /Users/username/.espressif/python_env/idf4.4_py3.10_env/bin/python ../../../components/esptool_py/esptool/esptool.py part to just python -m esptool).

Trying to update esptool in ESP-IDF:
It is recommended to not update esptool to an incompatible major release. However, if you really want to try, it is possible to try updating the esptool submodule used in IDF v4.4:

• Activate your ESP-IDF environment
• Navigate to the components/esptool_py/esptool/ directory
• Run git fetch and git reset origin/master --hard. This updates the esptool submodule to the latest version
• Run pip install . (still in the esptool directory). This installs the new esptool dependencies to the IDF environment
• Try building and flashing your app as usual
• To revert the esptool update, run git submodule update --init --recursive in the esp-idf directory.

Please note this is not the official way to do things (updates should be done by running the install script), but just a recommendation to try, since you've already started to hack into the ESP-IDF installation.