Python version problem with Eclipse IDE setup on macOS Big Sur

[mwirth]

Folks,

Trying to get the esp-idf plugin installed in the Eclipse IDE on a MacBook Pro (10,1; early 2013) running macOS Big Sur 11.0.1. Almost got there (as opposed to completely failing with the VS Code Onboarding; see separate post on that). But during the final stage of loading a new ESP-IDF project in Eclipse, the IDE insists on using Python 2.7 instead of 3.8.5 to run the code setup template, etc., and fails with complaints about a bunch of Python tools not installed in the 2.7 environment (vs the 3.8.5 environment, where they are installed.). See the console output below for details. Note that my current command shell environment is setup so that a simple "python ...." command invokes python 3.8.5.

Here are a couple MacOS Big Sur oddities that may aggravate the problem and Eclipse issues:

• The official path to python 2.7 is /usr/bin/python. Unfortunately, this is not (easily?) changeable, because that path is in the security-protected read-only part of the system.
• My path to python 3.8.5 is /usr/local/bin/python.
• I keep going to Help | ESP-IDF Tools Manager | Install Tools and resetting the Executable Locations for git and python from /user/bin/... to /usr/local/bin/..., but it doesn't seem to do any good. If I go through the whole "Install Tools" and then come back, the paths for git and python are reset to just /usr/bin/... again.
• If I execute Help | ESP-IDF Tools Manager | List Installed Tools menu item, the "Espressif IDF Tools Console" output shows that the wrong python is being used:

  Code: Select all

  Executing /usr/bin/python/Users/mike/esp/esp-idf/tools/idf_tools.py list
  * xtensa-esp32-elf: Toolchain for Xtensa (ESP32) based on GCC
    - esp-2020r3-8.4.0 (recommended, installed)
  * xtensa-esp32s2-elf: Toolchain for Xtensa (ESP32-S2) based on GCC
    - esp-2020r3-8.4.0 (recommended, installed)
  * xtensa-esp32s3-elf: Toolchain for Xtensa (ESP32-S3) based on GCC
    - esp-2020r3-8.4.0 (recommended, installed)
  * riscv-none-embed-gcc: Toolchain for RISC-V (optional)
    - riscv-none-embed-gcc-8.2.0 (recommended, installed)
  * esp32ulp-elf: Toolchain for ESP32 ULP coprocessor
    - 2.28.51-esp-20191205 (recommended, installed)
  * esp32s2ulp-elf: Toolchain for ESP32-S2 ULP coprocessor
    - 2.28.51-esp-20191205 (recommended, installed)
  * cmake: CMake build system (optional)
    - 3.16.4 (recommended, installed)
  * openocd-esp32: OpenOCD for ESP32
    - v0.10.0-esp32-20200709 (recommended, installed)
  * ninja: Ninja build system (optional)
    - 1.10.0 (recommended, installed)
  

• The wrong python also appears to be used when I create a New Espressif IDF Project. Here's the console output:

  Code: Select all

  Building in: /Users/mike/eclipse-workspace/BlinkyTest2/build
  cmake -G Ninja -DCMAKE_TOOLCHAIN_FILE=/Users/mike/esp/esp-idf/tools/cmake/toolchain-esp32.cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DIDF_TARGET=esp32 /Users/mike/eclipse-workspace/BlinkyTest2
  -- Unexpected file in components directory: /Users/mike/esp/esp-idf/components/.DS_Store
  -- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
  -- Building ESP-IDF components for target esp32
  -- Checking Python dependencies...
  The following Python requirements are not satisfied:
  click>=7.0
  pyelftools>=0.22
  gdbgui==0.13.2.0
  pygdbmi<=0.9.0.2
  kconfiglib==13.7.1
  reedsolo>=1.5.3,<=1.5.4
  bitstring>=3.1.6
  ecdsa>=0.16.0
  Please follow the instructions found in the "Set up the tools" section of ESP-IDF Getting Started Guide
  Diagnostic information:
      IDF_PYTHON_ENV_PATH: (not set)
      Python interpreter used: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
      Warning: python interpreter not running from IDF_PYTHON_ENV_PATH
      PATH: /Users/mike/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin:/Users/mike/.espressif/tools/xtensa-esp32s2-elf/esp-2020r3-8.4.0/xtensa-esp32s2-elf/bin:/Users/mike/.espressif/tools/xtensa-esp32s3-elf/esp-2020r3-8.4.0/xtensa-esp32s3-elf/bin:/Users/mike/.espressif/tools/riscv-none-embed-gcc/riscv-none-embed-gcc-8.2.0/xPacks/riscv-none-embed-gcc/8.2.0-3.1/bin:/Users/mike/.espressif/tools/esp32ulp-elf/2.28.51-esp-20191205/esp32ulp-elf-binutils/bin:/Users/mike/.espressif/tools/esp32s2ulp-elf/2.28.51-esp-20191205/esp32s2ulp-elf-binutils/bin:/Users/mike/.espressif/tools/cmake/3.16.4/CMake.app/Contents/bin:/Users/mike/.espressif/tools/openocd-esp32/v0.10.0-esp32-20200709/openocd-esp32/bin:/Users/mike/.espressif/tools/ninja/1.10.0/:/Users/mike/esp/esp-idf/tools:/usr/bin:/bin:/usr/sbin:/sbin;/usr/local/bin
  CMake Error at /Users/mike/esp/esp-idf/tools/cmake/build.cmake:272 (message):
    Some Python dependencies must be installed.  Check above message for
    details.
  Call Stack (most recent call first):
    /Users/mike/esp/esp-idf/tools/cmake/build.cmake:398 (__build_check_python)
    /Users/mike/esp/esp-idf/tools/cmake/project.cmake:396 (idf_build_process)
    CMakeLists.txt:6 (project)
  
  
  -- Configuring incomplete, errors occurred!
  See also "/Users/mike/eclipse-workspace/BlinkyTest2/build/CMakeFiles/CMakeOutput.log".
  cmake --build . -- -v
  ninja: error: loading 'build.ninja': No such file or directory
  Build complete (0 errors, 0 warnings): /Users/mike/eclipse-workspace/BlinkyTest2/build
  
  

How do I fix this (i.e., get both the plugin and Eclipse to use the right Python? I tried to do a workaround on my Mac to get /usr/bin/python to point to the 3.8.5 python, but as noted, this link is in the protected, read-only part of the system. Suggestions?

TIA,

Mike

PS: Many thanks for all the effort to supply IDEs for the ESP32.

[ESP_kondalkolipaka]

Hi @mwirth,

Sorry that it didn't work out of the box. It looks like the problem is with the IDF_PYTHON_ENV_PATH variable is not during the install tools and export variables process.

IDF_PYTHON_ENV_PATH is a virtual python environment path that is used during the application build process which is what is missing in your case.

Ideally, this is configured while installing tools from Eclipse. Help > ESP-IDF Tools Manager > Install Tools. Check the console once the install tools process is completed and look for IDF_PYTHON_ENV_PATH

In the console, check whether the following command executed successfully.

Code: Select all

Executing C:\Users\...python.exe C:\Users\...\esp\esp-idf\tools\idf_tools.py install-python-env

Check export command output, does it show IDF_PYTHON_ENV_PATH path?

Code: Select all

Executing C:\Users\..\python.exe C:\Users\..\esp-idf\tools\idf_tools.py export --format=key-value

It looks something like this IDF_PYTHON_ENV_PATH=C:\Users\..\.espressif\python_env\idf4.3_py3.7_env

If everything looks good so far, check whether the IDF_PYTHON_ENV_PATH is set to Eclipse CDT build environment variables(This is automatically taken care of)

Code: Select all

Preferences > C/C++ Build > Environment

If not set, could be some issue that needs a deeper look. However, you can set it manually as mentioned here https://github.com/espressif/idf-eclips ... -variables

The python path which is asked during the install tools doesn't really matter, which is used only to install tools, not after that. Every ESP-IDF has a compatible python environment which is configured by itself during the Install tools and which is used further during the build process.

I hope that helps. Please let us know your findings.

[mwirth]

kolipakakondal, thanks for your prompt reply. You're definitely right that the issue is which version of python gets executed at various points in the plugin installation process and the communication of version preferences between IDF and Eclipse. But macOS plays a major role here. Here's a summary of the issues, as I understand them:

• My macOS 11.0.1 Big Sur installation has two Python versions installed: 2.7.16 and 3.9.1, accessible via /usr/bin and /usr/local/bin, respectively. The binaries are in /Library/Frameworks/Python.framework/Versions/..., part of the system I also had version 3.8.5, installed by pyenv, in user space. Unfortunately, my pyenv setup was mangled -- not clear what python got used.
• Current macOS versions (e.g., Catalina and Big Sur) don't make this any easier. For example, I couldn't redirect /usr/bin/python to the 3.9.1 version to force it's use because that path is in protected system space, read-only.
• Quick question: I'm using z-shell and have my initialization for command line executions in the .zshrc file. Is that file executed by Eclipse at startup, i.e., can I depend on environment variables set in it?
• So I uninstalled Python 3.8.5 and all of pyenv, to remove those complications. The Current alias at .../Python.framework/Versions/Current is set to the 3.9.1 version (quite new, but desirable because it fixes a bunch of Big Sur-related bugs)
• After this cleanup, I reinstalled Eclipse and the ESP-IDF plugin. Here's a sequence of screen shots from that process that I hope give you the information need to diagnose my problem.
• Here's a random startup error. Presume it isn't critical to the issue at hand.

  

  Eclipse IDF 1.png (31.86 KiB) Viewed 24744 times

• Note the settings here. The defaults provided for the locations of git and python are in /usr/bin/. I keep changing that to /usr/local/bin/..., but it doesn't stick. Is that a bug? Is this where the env. variable for Python's location is set within the plugin?

Having trouble adding more images. Will keep adding Replies until all 5 are added.

[mwirth]

<Continuation of reply, starting with screenshot 2>

• Note the settings here. The defaults provided for the locations of git and python are in /usr/bin/. I keep changing that to /usr/local/bin/..., but it doesn't stick. Is that a bug? Is this where the env. variable for Python's location is set within the plugin?

  

  Eclipse IDF 2.png (43.2 KiB) Viewed 24744 times

• Console output from installation of tools:

  

  Eclipse IDE 3.png (187.93 KiB) Viewed 24744 times

• Note that git is available at /usr/local/bin/git as entered in the tools dialog. (Python 3.9.1 was checked, too.)

  

  Eclipse IDF 4.png (31.6 KiB) Viewed 24744 times

<One more screenshot to go -- the critical one! See next reply.>

[mwirth]

<Final Image>

• Here's the screenshot from trying to build a new IDF project, BlinkyTest2. Note that IDF_PYTHON_ENV_PATH isn't set. Is this the critical error? That is, Python 2.7, somehow being used, instead of 3.9.1, so the associated Python requirements aren't satisfied.? How do I set this environment variable? Can I explicitly do it in .zshrc? Somewhere else?

  

  Eclipse IDE 5.png (532.58 KiB) Viewed 24744 times

Thanks!

Mike

PS: Some general clarification of these issues in the documentation would be useful. If the new read-only protections of the system image in macOS Catalina and Big Sur create extra issues, that would benefit from documentation as well.

[ESP_kondalkolipaka]

Hi Mike,

Did you have a chance to set IDF_PYTHON_ENV_PATH to python path as suggested in the previous post? This you can set via

Code: Select all

Eclipse Preferences > C/C++ Build > Environment

Please rebuild the project once you set the IDF_PYTHON_ENV_PATH to the python path and see if that helps.

This link could help
https://github.com/espressif/idf-eclips ... -variables

Let me reply again after analyzing the screenshots and information you've provided.

[mwirth]

kolipakakondal,

I tried lots of experiments to no avail:

1. Set the IDF_PYTHON_ENV_PATH environment variable, as in:

   

   Eclipse build env prefs.png (162.64 KiB) Viewed 24618 times

2. I installed the plugin from a local archive.
3. I installed the plugin from the Eclipse Marketplace.
4. I poked around the Python code in the Espressif tools and placed some diagnostic print statement there, but finally realized the relevant code was Java code in your repo (i.e., where the environment variables are tested and Python connections made).

In all cases, I get the same result: When I attempt to build a sample project, it fails with:

1. The Python environment actually used is not 3.9.1.
2. A bunch of Python requirements are not satisfied -- because of the wrong Python environment.
3. IDF_PYTHON_ENV_PATH was either NULL (in my earlier experiments) or /Users/mike/.espressif/python_env/idf4.2_py3.9_env now.
4. No matter, the Python interpreter used is still /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

Here's the exact console output:

Building in: /Users/mike/eclipse-workspace/BlinkyTest2/build
cmake -G Ninja -DCMAKE_TOOLCHAIN_FILE=/Users/mike/esp/esp-idf/tools/cmake/toolchain-esp32.cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DIDF_TARGET=esp32 /Users/mike/eclipse-workspace/BlinkyTest2
-- Unexpected file in components directory: /Users/mike/esp/esp-idf/components/.DS_Store
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32
-- Checking Python dependencies...
The following Python requirements are not satisfied:
click>=7.0
pyelftools>=0.22
gdbgui==0.13.2.0
pygdbmi<=0.9.0.2
kconfiglib==13.7.1
reedsolo>=1.5.3,<=1.5.4
bitstring>=3.1.6
ecdsa>=0.16.0
To install the missing packages, please run "/Users/mike/esp/esp-idf/install.sh"
Diagnostic information:
IDF_PYTHON_ENV_PATH: /Users/mike/.espressif/python_env/idf4.2_py3.9_env
Python interpreter used: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Warning: python interpreter not running from IDF_PYTHON_ENV_PATH
PATH: /Users/mike/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin:/Users/mike/.espressif/tools/xtensa-esp32s2-elf/esp-2020r3-8.4.0/xtensa-esp32s2-elf/bin:/Users/mike/.espressif/tools/xtensa-esp32s3-elf/esp-2020r3-8.4.0/xtensa-esp32s3-elf/bin:/Users/mike/.espressif/tools/riscv-none-embed-gcc/riscv-none-embed-gcc-8.2.0/xPacks/riscv-none-embed-gcc/8.2.0-3.1/bin:/Users/mike/.espressif/tools/esp32ulp-elf/2.28.51-esp-20191205/esp32ulp-elf-binutils/bin:/Users/mike/.espressif/tools/esp32s2ulp-elf/2.28.51-esp-20191205/esp32s2ulp-elf-binutils/bin:/Users/mike/.espressif/tools/cmake/3.16.4/CMake.app/Contents/bin:/Users/mike/.espressif/tools/openocd-esp32/v0.10.0-esp32-20200709/openocd-esp32/bin:/Users/mike/.espressif/tools/ninja/1.10.0/:/Users/mike/esp/esp-idf/tools:/usr/bin:/bin:/usr/sbin:/sbin;/usr/local/bin
CMake Error at /Users/mike/esp/esp-idf/tools/cmake/build.cmake:272 (message):
Some Python dependencies must be installed. Check above message for
details.
Call Stack (most recent call first):
/Users/mike/esp/esp-idf/tools/cmake/build.cmake:398 (__build_check_python)
/Users/mike/esp/esp-idf/tools/cmake/project.cmake:396 (idf_build_process)
CMakeLists.txt:6 (project)


-- Configuring incomplete, errors occurred!
See also "/Users/mike/eclipse-workspace/BlinkyTest2/build/CMakeFiles/CMakeOutput.log".
cmake --build . -- -v
ninja: error: loading 'build.ninja': No such file or directory
Build complete (0 errors, 0 warnings): /Users/mike/eclipse-workspace/BlinkyTest2/build

If it' would be useful, I can poke around in your java code, especially where the IDF_PYTHON_ENV_PATH env. variables evaluated and the connection to the Python interpreter is done. Just let me know where to start in the java code, e.g., what classes.

Thanks,

Mike

[seanmbates@gmail.com]

Mike,
I have the same problem. Did you find a solution to this issue?

Thanks,
Sean

[mwirth]

Sean,

No, have not found a solution for this specific problem. But I have encountered a version of it again in the IDE setup for yet another brand of SBCs. In both cases, Python is actively used in the IDE setup and application build chain. It's clear that differences in the Python environment (version used, environment variables, packages installed or not, package versions) can cause multiple obscure errors. It's made more complicated with macOS, where working around the pre-installed Python versions in the read-only system kernel adds difficulties.

As a result, I've had to resort to using virtual environment tools (e.g., venv at the Python level, asdf as a higher level, multi-language tool) to create a specific execution environment as if the IDE were a fresh install on a clean machine.

I haven't gotten back to doing this for the ESP32 IDE. Perhaps you or someone familiar with setting up Python virtual execution environments on the Mac can give it a try.

Another thing that would be useful is if someone on the IDE development team or an experienced user could document additional validation tests that could be used after each IDE installation step to verify its correct result.

HTH,

Mike (now running macOS Monterey)