Upgrading ESP-IDF from v4.4.7 to v5.0

[aapee-jcv]

Hi!

I am using Ubuntu 22.04.5 LTS (Jammy Jellyfish) and python3-venv is installed, however I get following error:

Code: Select all

Extracting /opt/esp/dist/cmake-3.24.0-linux-x86_64.tar.gz to /opt/esp/tools/cmake/3.24.0
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, [b]you need to install the python3-venv[/b]
package using the following command.

    apt install python3.8-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/opt/esp/python_env/idf5.0_py3.8_env/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']

Creating a new Python environment in /opt/esp/python_env/idf5.0_py3.8_env
Traceback (most recent call last):
  File "/opt/esp/idf/tools/idf_tools.py", line 2529, in <module>
    main(sys.argv[1:])
  File "/opt/esp/idf/tools/idf_tools.py", line 2521, in main
    action_func(args)
  File "/opt/esp/idf/tools/idf_tools.py", line 1997, in action_install_python_env
    subprocess.check_call([sys.executable, '-m', 'venv',
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-m', 'venv', '--clear', '/opt/esp/python_env/idf5.0_py3.8_env']' returned non-zero exit status 1.
Error: error building at STEP "RUN :   && update-ca-certificates --fresh   && $IDF_PATH/tools/idf_tools.py --non-interactive install required --targets=${IDF_INSTALL_TARGETS}   && $IDF_PATH/tools/idf_tools.py --non-interactive install cmake   && $IDF_PATH/tools/idf_tools.py --non-interactive install-python-env   && rm -rf $IDF_TOOLS_PATH/dist   && :": error while running runtime: exit status 1

Same happens with tags v5.0 and v.5.2 so it seems quite systematic after version 5.0 and higher.

[ESP_Roland]

Have you run the suggested "apt install python3.8-venv" command?

python3-venv is an alias of python3.10-venv in your system (https://packages.ubuntu.com/jammy/python3-venv). And based on your logs, our installer picked up 3.8 instead. Maybe you made 3.8 the default one? See the output of "python3 --version".

[aapee-jcv]

Code: Select all

$ python3 --version
Python 3.10.12

$ apt list --installed|grep venv
python3-venv/jammy-updates,now 3.10.6-1~22.04.1 amd64 [installed]
python3.10-venv/jammy-updates,jammy-security,now 3.10.12-1~22.04.6 amd64 [installed,automatic]

That specific version is not found, so though never on should do:

Code: Select all

$ sudo apt install python3.8-venv
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package python3.8-venv
E: Couldn't find any package by glob 'python3.8-venv'

[ESP_Roland]

Ok, I see. Python 3.10 would be better but I don't understand why it is not picked up. python3 is tested before python3.8.

Or your "python --version" is of 3.8?

[aapee-jcv]

I have downgraded python to 3.8 and made it default system wide, and virtualenv is installed with pip.

$ python3 --version
Python 3.8.2
$ python --version
Python 3.8.2
$ python3 -m pip list
Package Version
------------ -------
distlib 0.3.9
filelock 3.16.1
pip 24.3.1
platformdirs 4.3.6
setuptools 41.2.0
virtualenv 20.27.1

Command I am running: (just wondering could this problem lead to Dockerfile and entrypoint.sh used?)
docker build -t expressif-env-v5.2 . --build-arg IDF_CLONE_BRANCH_OR_TAG=v5.2

I am still wondering why it ends to same error...

Creating a new Python environment in /opt/esp/python_env/idf5.2_py3.8_env
Traceback (most recent call last):
File "/opt/esp/idf/tools/idf_tools.py", line 2806, in <module>
main(sys.argv[1:])
File "/opt/esp/idf/tools/idf_tools.py", line 2798, in main
action_func(args)
File "/opt/esp/idf/tools/idf_tools.py", line 2214, in action_install_python_env
subprocess.check_call([sys.executable, '-m', 'venv',
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-m', 'venv', '--clear', '/opt/esp/python_env/idf5.2_py3.8_env']' returned non-zero exit status 1.
Error: error building at STEP "RUN : && update-ca-certificates --fresh && $IDF_PATH/tools/idf_tools.py --non-interactive install required --targets=${IDF_INSTALL_TARGETS} && $IDF_PATH/tools/idf_tools.py --non-interactive install cmake && $IDF_PATH/tools/idf_tools.py --non-interactive install-python-env && rm -rf $IDF_TOOLS_PATH/dist && :": error while running runtime: exit status 1

[ESP_Roland]

Virtualenv installed by pip is not used. The Venv is an internal module of Python. For some reason Debian/Ubuntu and derivate operating systems divide them up and the Venv module is placed in the packages ending with "-venv".

So if your python3.8 is missing venv then I don't think you can fix it. Certainly not by installing virtualenv.

[ESP_Roland]

Do you know that Espressif provides docker images with ESP-IDF (https://hub.docker.com/r/espressif/idf/tags)? Here are the sources for v5.2: https://github.com/espressif/esp-idf/bl ... Dockerfile