"ModuleNotFoundError: No module named 'chardet'" during build on MacOS M1.

[corneo]

I'm running into a problem using ESP-IDF on M1 mac. I've tried everything I've found to resolve it, but obviously, I have had no luck. The crux is that the "chardet" python module is missing. All the similar problems I have read about involving missing 'chardet' come down to the requirement to install Rosetta. I have done that. The softwareupdate command does report failure, but feedback on that says it's not a problem and to ignore it. (I'm skeptical.)

esolving esp32dev dependencies...
Already up-to-date.
Updating metadata for the vscode IDE...
Error: Processing esp32dev (platform: espressif32; board: esp32dev; framework: espidf)
--------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards ... 32dev.html
PLATFORM: Espressif 32 (6.1.0) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
- framework-espidf @ 3.50001.0 (5.0.1)
- tool-cmake @ 3.16.4
- tool-esptoolpy @ 1.40500.0 (4.5.0)
- tool-ninja @ 1.9.0
- toolchain-esp32ulp @ 1.23500.220830 (2.35.0)
- toolchain-xtensa-esp32 @ 11.2.0+2022r1
Reading CMake configuration...
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32
-- Configuring incomplete, errors occurred!
See also "/Users/tim/Library/CloudStorage/Dropbox/tim/Projects/Electronics/PlatformIO/Projects/WiFiManager-sandbox/.pio/build/esp32dev/CMakeFiles/CMakeOutput.log".

fatal: not a git repository (or any of the parent directories): .git
CMake Error at /Users/tim/.platformio/packages/framework-espidf/tools/cmake/build.cmake:519 (message):
Traceback (most recent call last):

File "/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/requests/compat.py", line 11, in <module>
import chardet

ModuleNotFoundError: No module named 'chardet'



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/idf_component_manager/prepare_components/__main__.py", line 4, in <module>
from .prepare import main
File "/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/idf_component_manager/prepare_components/prepare.py", line 18, in <module>
from ..core import ComponentManager
File "/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/idf_component_manager/core.py", line 17, in <module>
import requests
File "/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/requests/__init__.py", line 45, in <module>
from .exceptions import RequestsDependencyWarning
File "/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/requests/exceptions.py", line 9, in <module>
from .compat import JSONDecodeError as CompatJSONDecodeError
File "/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/requests/compat.py", line 13, in <module>
import charset_normalizer as chardet
File "/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/charset_normalizer/__init__.py", line 24, in <module>
from .api import from_bytes, from_fp, from_path
File "/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/charset_normalizer/api.py", line 5, in <module>
from .cd import (
File "/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/charset_normalizer/cd.py", line 9, in <module>
from .md import is_suspiciously_successive_range

ImportError:
dlopen(/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/charset_normalizer/md.cpython-311-darwin.so,
0x0002): tried:
'/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/charset_normalizer/md.cpython-311-darwin.so'
(mach-o file, but is an incompatible architecture (have 'arm64', need
'x86_64')),
'/System/Volumes/Preboot/Cryptexes/OS/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/charset_normalizer/md.cpython-311-darwin.so'
(no such file),
'/Users/tim/.platformio/penv/.espidf-5.0.1/lib/python3.11/site-packages/charset_normalizer/md.cpython-311-darwin.so'
(mach-o file, but is an incompatible architecture (have 'arm64', need
'x86_64'))

Call Stack (most recent call first):
/Users/tim/.platformio/packages/framework-espidf/tools/cmake/project.cmake:440 (idf_build_process)
CMakeLists.txt:3 (project)



========================== [FAILED] Took 1.03 seconds ==========================

How can I work around this? [short of running windows or using a non-M1 mac. (;->)]

[corneo]

Darn... crickets.

[larboyer]

I believe I've finally solved this issue. When you install Espressif on mac m1 machines (and maybe elsewhere) it installs it's own version of Python. For me this was located under the 'hidden' espressif dir
~/.espressif/python_env/idf5.2_py3.9_env/lib/python3.9/site-packages.
However, for some reason, they forgot to install that 'chardet' package.

When I would manually do a 'pip install chardet' it would install properly, but in the _mac_ python installation, which, for me, was located:
~/Library/Python/3.9/lib/python/site-packages

To solve, this, find out where pip puts chardet
pip show chardet
Then go to that directory and then copy it to the espressif directory, something like
cd ~/Library/Python/3.9/lib/python/site-packages
cp -R chardet ~/.espressif/python_env/idf5.2_py3.9_env/lib/python3.9/site-packages

Hope this helps some other dev out there not loose 6 hours like me ;>)

[hcamacho]

Here is a way to solve this problem directly. If you look at the error, just above you will see a path for the venv:

Code: Select all

CMake Error at /Users/xxxxxx/esp/esp-idf/tools/cmake/build.cmake:544 (message):
  Traceback (most recent call last):

    File "/Users/xxxxx/.espressif/python_env/idf5.2_py3.9_env/lib/python3.9/site-packages/requests/compat.py", line 11, in <module>
      import chardet

  ModuleNotFoundError: No module named 'chardet'

Perform the following:

Code: Select all

% cd /Users/xxxxxx/.espressif/python_env
% ls
idf5.2_py3.9_env

% cd idf5.2_py3.9_env/bin
% . ./activate
(idf5.2_py3.9_env) % pip install chardet
Collecting chardet
  Downloading chardet-5.2.0-py3-none-any.whl.metadata (3.4 kB)
Downloading chardet-5.2.0-py3-none-any.whl (199 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.4/199.4 kB 3.8 MB/s eta 0:00:00
Installing collected packages: chardet
Successfully installed chardet-5.2.0

This will install chardet directly to the venv that esp32-idf is using.

HFC