velxio/docs/wiki/esp32-external-library-compilation.md

ESP32 External Arduino Library Compilation — IDF Component Approach

Scope: This document covers the full investigation and implementation of automatic external Arduino library inclusion when compiling ESP32 sketches via ESP-IDF. Target audience: future maintainers who need to understand why the library resolution works the way it does and what bugs were discovered during real compilation testing.

---

Table of Contents

1. Problem Statement
2. Root Cause
3. Solution Architecture
4. Implementation Details
5. Bugs Found During Real Compilation
6. Error Visibility Improvements
7. Test Coverage
8. Files Changed

---

Problem Statement

When compiling an ESP32 sketch that uses an external library (e.g. the DHT22 temperature sensor example with #include <DHT.h>), the build failed with:

fatal error: DHT.h: No such file or directory

For Arduino UNO, arduino-cli automatically scans ~/Arduino/libraries/ and adds the correct -I flags. For ESP32 via ESP-IDF, that path is never scanned — ESP-IDF only knows about components explicitly listed in EXTRA_COMPONENT_DIRS.

Additionally, even when an error did occur, it was invisible in the UI: ninja build errors go to stdout (not stderr), so the frontend was classifying them as grey info lines instead of red error lines.

---

Root Cause

ESP-IDF uses a CMake-based component system. Every unit of code must be a registered IDF component with its own CMakeLists.txt calling idf_component_register(). The arduino-esp32 library itself is included this way. External Arduino libraries (installed via arduino-cli Library Manager into ~/Arduino/libraries/) have no idf_component_register() and are therefore invisible to the build system.

The naive fix of copying .h/.cpp files flat into main/ breaks libraries that use subdirectory structures for internal includes (e.g. #include "utility/xyz.h").

---

Solution Architecture

Each external Arduino library is wrapped as a proper ESP-IDF component:

project/
  CMakeLists.txt              ← adds user_libs/ to EXTRA_COMPONENT_DIRS
  user_libs/
    DHT_sensor_library/
      CMakeLists.txt          ← idf_component_register(SRCS ... REQUIRES arduino-esp32)
      DHT.h
      DHT.cpp
      DHT_U.h
      DHT_U.cpp
    Adafruit_Unified_Sensor/  ← transitive dependency, auto-discovered
      CMakeLists.txt
      Adafruit_Sensor.h
      Adafruit_Sensor.cpp
  main/
    CMakeLists.txt            ← REQUIRES arduino-esp32 DHT_sensor_library Adafruit_Unified_Sensor
    main.cpp
    sketch.ino.cpp

ESP-IDF automatically compiles every subdirectory of EXTRA_COMPONENT_DIRS that contains an idf_component_register() call. The INCLUDE_DIRS "." inside each component makes its headers available to anything that REQUIRES it.

Library Search Order

1. $ARDUINO_ESP32_PATH/libraries/ — ESP32-native libs bundled with arduino-esp32 (WiFi, BLE, EEPROM…) — already compiled as part of arduino-esp32 component, so usually skipped
2. ~/Documents/Arduino/libraries/ — user-installed libraries (Windows primary path)
3. ~/Arduino/libraries/ — alternate user path
4. /root/Arduino/libraries/ — Docker / CI root user path

---

Implementation Details

_detect_external_includes(code)

Scans source code for #include <Header.h> directives and returns those that are NOT:

• Arduino/ESP32 built-ins (e.g. Arduino.h, Wire.h, WiFi.h, esp_wifi.h)
• Headers with / (ESP-IDF internal paths like freertos/FreeRTOS.h)
• ESP-IDF pattern prefixes (esp_, driver/, soc/, hal/, nvs, rom/)

_find_library_for_header(header, libs_dir)

Iterates subdirectories of libs_dir, checking both the root and src/ subdirectory for the requested header. Returns the source root (either lib_dir/ or lib_dir/src/).

_create_idf_component(header, src_root, user_libs_dir, arduino_comp_name)

Creates user_libs/<safe_name>/ with:

• All .h, .cpp, .c files copied flat from src_root
• A generated CMakeLists.txt:

  idf_component_register(
      SRCS "DHT.cpp" "DHT_U.cpp"
      INCLUDE_DIRS "."
      REQUIRES arduino-esp32
  )
  

• Returns the component directory name (e.g. DHT_sensor_library)

Transitive Dependency Resolution (BFS)

The library resolution loop uses breadth-first search:

1. Phase 1 — BFS discovery: Start with headers found in the user sketch. After creating each component, scan its copied .h files for further external includes. Enqueue any new headers not yet resolved. Repeat until the queue is empty. A header_to_comp dict tracks which component provides each header.

2. Phase 2 — inter-component REQUIRES: After all components are created, scan each component's headers again. For any dependency that maps to another component in header_to_comp, patch that component's CMakeLists.txt to add the dep to REQUIRES.

This ensures DHT_sensor_library/CMakeLists.txt ends up with:

REQUIRES arduino-esp32 Adafruit_Unified_Sensor

Main CMakeLists.txt Patching

The template main/CMakeLists.txt uses a CMake variable:

REQUIRES ${_arduino_comp_name}

The Python patch looks for this exact string (not the resolved literal arduino-esp32) and appends the user library component names:

REQUIRES ${_arduino_comp_name} DHT_sensor_library Adafruit_Unified_Sensor

---

Bugs Found During Real Compilation

Three bugs were discovered when running an actual ESP32 compile (vs. unit tests with mocks):

Bug 1 — Wrong Component Name (libraries instead of DHT_sensor_library)

Symptom: Build step showed esp-idf/libraries/CMakeFiles/... instead of esp-idf/DHT_sensor_library/CMakeFiles/....

Cause: _create_idf_component used src_root.parent.name to get the library name. When _find_library_for_header returns the library root (no src/ subdir), src_root is the library directory, so .parent.name gives libraries (the parent search dir).

Fix:

# Before
lib_dir_name = src_root.parent.name

# After
lib_dir_name = src_root.parent.name if src_root.name == 'src' else src_root.name

Bug 2 — Missing Transitive Dependency (Adafruit_Sensor.h)

Symptom: After fixing Bug 1, the build failed with:

DHT_U.h:36:10: fatal error: Adafruit_Sensor.h: No such file or directory

Cause: DHT_U.h (part of DHT library) #includes Adafruit_Sensor.h from the Adafruit_Unified_Sensor library. The original code only scanned the user sketch for external includes, not the library headers themselves.

Fix: Added BFS transitive dependency resolution (Phase 1 + Phase 2 described above).

Bug 3 — CMake Template Variable Mismatch

Symptom: After fixing Bugs 1 and 2, the main component still couldn't find DHT.h:

sketch.ino.cpp:3:10: fatal error: DHT.h: No such file or directory

Cause: The Python patch looked for REQUIRES arduino-esp32 (literal) in main/CMakeLists.txt, but the template uses REQUIRES ${_arduino_comp_name} (CMake variable). The replacement was silently a no-op.

Fix: Updated the patch to match the CMake variable syntax:

for old_req in [r'REQUIRES ${_arduino_comp_name}', f'REQUIRES {arduino_comp_name}']:
    if old_req in cmake_text:
        cmake_text = cmake_text.replace(old_req, f'{old_req} {main_reqs}')
        break

---

Error Visibility Improvements

Two frontend/backend changes were made to ensure compilation errors are clearly visible:

Backend — Ninja Error Extraction

When ESP-IDF/ninja fails, compiler errors go to stdout (not stderr). The backend now detects FAILED: blocks in stdout and moves them to the stderr field of the response, so the frontend classifies them as errors:

if stripped.startswith('FAILED:') or stripped == 'ninja: build stopped: subcommand failed.':
    in_failed_block = True
    # ... extract and move to stderr

Frontend — Compilation Console Auto-Filter

CompilationConsole.tsx now:

• Tracks previous log count with prevLogsLenRef
• Detects newly arrived error logs via useEffect
• Automatically switches the filter to "Errors" view when new errors arrive

compilationLogger.ts classifies stdout lines in ninja FAILED: blocks as 'error' type (not 'info'), using a state machine (inFailedBlock flag).

---

Test Coverage

backend/test_espidf_compiler.py — 25 unit tests, no ESP-IDF toolchain required:

Class	Tests	What it covers
TestDetectExternalIncludes	8	DHT.h detected; Arduino.h, Wire.h, esp_* skipped; path headers skipped
TestFindLibraryForHeader	4	Root layout, src/ layout, missing library, empty dir
TestCreateIdfComponent	11	Dir created, CMakeLists.txt content, files copied, name sanitization, correct library name
TestTemplateCMakeLists	2	Template files contain user_libs block and REQUIRES placeholder

Run from backend/:

python test_espidf_compiler.py

---

Files Changed

File	Change
backend/app/services/espidf_compiler.py	Added _detect_external_includes, _find_library_for_header, _create_idf_component; BFS transitive dep resolution; CMake patching fixes
backend/app/services/esp-idf-template/CMakeLists.txt	Added user_libs/ to EXTRA_COMPONENT_DIRS via EXISTS guard
frontend/src/utils/compilationLogger.ts	Ninja FAILED: block state machine; classifies lines as 'error'
frontend/src/components/editor/CompilationConsole.tsx	Auto-switches to Errors filter when new errors arrive
backend/test_espidf_compiler.py	25-test suite covering all library resolution logic