Two-tier rule replacing the blanket skip:
- 'all' now counts as wildcard ('*'): LiquidCrystal_I2C 2.0.0 declares
architectures=all and was skipped on every ESP32 build, breaking the
most popular I2C LCD library with a bare 'No such file or directory'.
- Headers the SKETCH includes directly merge even when the library
declares a foreign-only architecture (many avr-declared libs are pure
Wire/SPI code that compiles fine; a truthful compile error beats a
missing-header one). Transitive pulls keep the HARD guard, which is
the path that dragged SAMD-only Adafruit_ZeroDMA into ESP32 builds.
Adds a third entry to the board language selector next to Arduino C++
and MicroPython: ESP-IDF. In this mode the user writes a plain ESP-IDF
project — app_main() entry point, FreeRTOS + driver APIs — and the
backend compiles it through the same ESP-IDF toolchain it already uses
for ESP32 Arduino sketches, just without the arduino-esp32 component.
Backend:
- CompileRequest.language ('espidf') threaded through the sync + async
compile paths and folded into the dedup job key (language='arduino'
and omitted hash identically so old clients keep dedupping).
- espidf_compiler: pure_idf flag. User files are written into main/
as-is (no Arduino.h wrap, no velxio_compat.h, Arduino library
resolution skipped), ARDUINO_ESP32_PATH is dropped from the build env
and VELXIO_PURE_SKETCH raised so the template CMake compiles the
user's own sources via a glob branch. Pure builds get their own
persistent build-dir variant through the eff_hash fold.
- QEMU WiFi compat for IDF-style code: esp_wifi.h/esp_wifi_init
detection sets has_wifi, and literal #define SSID/PASS plus
wifi_config_t designated initializers are normalized to the QEMU AP.
- CONFIG_ARDUINO_* lines are stripped from sdkconfig.defaults in pure
mode (the symbols don't exist without the arduino component).
Frontend:
- LanguageMode gains 'espidf'; BOARD_SUPPORTS_ESPIDF covers the ESP32
family (Xtensa, S3, C3). Toolbar shows the option only for those.
- Switching modes seeds a main.c blink skeleton (app_main + gpio
driver), mirroring the MicroPython main.py flow.
- compileCode sends language='espidf'; run/stop paths are unchanged
(the QEMU worker consumes the same merged flash image).
- New gallery example: esp32-idf-blink (LED + resistor on GPIO 2).
Tests: unit coverage for the build-env switch, IDF wifi normalization,
job-key variance, file-group seeding and the new example; verified
end-to-end in a container from the prod image (pure build produces a
bootable flash image; Arduino-mode build unchanged, same variant hash).
_idf_target had no S3 case, so every S3 FQBN (esp32s3 / XIAO_ESP32S3 /
nano_nora) compiled as plain esp32 (LX6) and could not boot the S3 QEMU
machine. Add _is_esp32s3 (covers all three FQBNs; nano_nora has no 's3'
token) + return 'esp32s3'; add the xtensa-esp32s3-elf toolchain to the
build PATH; place the S3 second-stage bootloader at flash offset 0x0
(like C3) instead of 0x1000 via 'is_c3 or idf_target==esp32s3' at the merge.
Requires the xtensa-esp32s3-elf toolchain in the image
(install.sh esp32,esp32c3,esp32s3) and a libqemu-xtensa with an
esp32s3(-picsimlab) machine.
arduino-esp32's WiFiClientSecure/ssl_client.cpp wraps its ENTIRE body
(start_ssl_client, ssl_init, send_ssl_data, ...) in
#if !defined(MBEDTLS_KEY_EXCHANGE_SOME_PSK_ENABLED) ... #else <body> #endif
ESP-IDF's mbedtls defaults the PSK key-exchange modes OFF, so the object
compiled empty and any sketch using WiFiClientSecure — including
HTTPClient.begin(url), which links the secure client even for http:// —
failed to link with "undefined reference to start_ssl_client". Commenting
out begin() let the optimizer drop the unused client, which is why it
"compiled when commented".
- sdkconfig.defaults.in: enable the PSK key-exchange ciphersuites
(CONFIG_MBEDTLS_PSK_MODES + the four KEY_EXCHANGE_*PSK), matching
arduino-esp32's own sdkconfig.
- espidf_compiler: ESP-IDF only seeds sdkconfig from sdkconfig.defaults when
sdkconfig is ABSENT. Persistent build dirs live in the build volume and
keep a stale sdkconfig across image rebuilds, so the new CONFIG_* would
never reach kconfig. Drop the generated sdkconfig when the rendered
defaults change so it re-seeds on configure.
- test: assert the rendered sdkconfig enables PSK.
Verified end-to-end: the reported WiFi+HTTPClient sketch now compiles to a
1.1 MB binary (was a hard link error before).
The last paths that read the shared global /root/Arduino/libraries: a compile
with NO manifest (libraries=null — any from-scratch/anon sketch; the manifest is
never auto-derived from #includes) and the incomplete-manifest scan-all retry
(which re-enters compile unscoped). Both fell through to the global dir,
bypassing the cache entirely (a cached lib still failed when global was gone).
Now, when no scope is materialized, point the library search at the cache: the
cache root is itself a valid Arduino libraries dir (each <name@ver-sha> child is
a library), exposed via env (pro overlay sets them):
- arduino-cli: ARDUINO_DIRECTORIES_USER = VELXIO_FALLBACK_SKETCHBOOK (whose
libraries/ -> cache root) when scope_dir is None.
- ESP-IDF: _find_arduino_libraries_dir() prefers VELXIO_FALLBACK_LIBRARIES_DIR.
Unset (OSS self-host) -> legacy global, unchanged.
Also strip a trailing @version from manifest names (_bare_lib_names): norm_name
fused 'ArduinoJson@6.21.5' -> 'arduinojson6215' (cache miss -> global scan-all);
the per-board boards_json manifest is the path that still carries @version.
So a scoped compile can resolve the project OWNER's per-user custom libraries
(not the requester's) — a shared/embed/anon compile of someone else's project
still finds that owner's uploaded libs.
- core/hooks.py: new get_project_owner hook; materialize_library_scope gains an
opaque owner_id param (no-op default unchanged).
- espidf_compiler.compile/_attempt: thread owner_id to the materializer.
- compile.py: resolve owner via get_project_owner(project_id), pass to compile.
Additive: the OSS image (no overlay) ignores owner_id; index libs still resolve
from the cache. Foundation for per-user custom-lib storage (P2.2a write side).
A scoped ESP32 compile can now resolve libraries from a per-compile directory
provided by an overlay (the manifest's libs symlinked from a content-addressed
cache, with a legacy-dir fallback) instead of the single shared global volume.
- core/hooks.py: register_materialize_library_scope / materialize_library_scope
(no-op default -> None, so the OSS image keeps its single scan-all dir).
- espidf_compiler: _attempt(allowed) calls the hook, folds the returned content
token into the build-variant eff_hash (a content change gets a clean build
dir), passes libraries_dir to _compile_in_dir (arduino_libs = libraries_dir or
_find_arduino_libraries_dir()), and removes the throwaway dir after. The
graceful scan-all fallback (allowed=None) keeps using the default dir, so the
worst case of any materializer failure is fall-back-to-legacy (no break).
Per-variant build dirs fixed the cross-compile staleness, but the FIRST build of
a cold variant can still occasionally hit ESP-IDF nested-build flakiness (cmake /
bootloader / managed-components / sdkconfig). These are infrastructure failures,
never user code, and clear on a retry once the variant dir is warmer. Retry the
attempt once when the failure matches infrastructure markers (not a user-sketch
or missing-library error). Cheap via ccache + ninja incremental.
Replaces the wipe-on-change approach (which left ESP-IDF's nested bootloader /
managed-components build in a broken state under rapid reconfigure -
intermittent 'managed_components_list.temp.cmake: No such file'). Each distinct
configuration (board options x resolved library set, via the variant key the
caller already computes) now gets its OWN persistent project dir with its OWN
build/, never wiped or reconfigured for a different config:
- same config -> same dir -> warm ninja incremental + ccache (fast iterate);
- different config -> different dir -> isolated, consistent, no staleness,
no nested-build breakage;
- the scoped vs scan-all fallback attempts land in different dirs, so the
double-compile no longer corrupts a shared build/.
Variant dirs are LRU-bounded (_MAX_BUILD_VARIANTS per target); the global ccache
warms a fresh/evicted variant in seconds. Cleans up the old single-project
layout on first run. Fixes the cross-compile staleness for legacy AND manifest
compiles, and the fallback regression.
Supersedes the mid-_compile_in_dir build/ wipe (161deb9): wiping build/ AFTER
materializing libs but right before cmake left ESP-IDF's config half-regenerated
during the fallback's scoped->scan-all double-compile, intermittently failing
with 'sdkconfig.h: No such file'.
Instead fold the effective library set (the manifest, else the sketch's
non-core external includes) into the per-attempt build-dir hash, so a changed
lib set — or the scan-all fallback after a scoped attempt — resets the
persistent build/ at _prepare_persistent_project_dir time (before any cmake).
That is the existing, well-tested early-wipe path, so the configure is always
clean. Same lib set across compiles keeps the warm ccache/ninja cache; core-only
sketches share one dir (core headers filtered out of the token) so they never
trigger a spurious wipe.
Fixes both the original cross-compile staleness (intermittent cmake-configure
failures + stale-object false positives) and the fallback regression.
The persistent per-target build/ caches ESP-IDF's cmake configuration, ninja's
incremental graph and ccache-backed objects, all assuming a stable component
set. When consecutive compiles on the same dir have a DIFFERENT resolved
user_libs set (a different project/user, or a different library manifest) that
cache is inconsistent and produces two real failures:
- cmake reconfigure intermittently fails ('cmake configure failed') even
though each manifest compiles fine on a clean dir;
- ninja/ccache reuse a previous compile's objects/headers, letting a
now-absent library slip through as a false-positive success against a lib
the current sketch/manifest no longer includes.
Fingerprint the materialized user_libs/ (sorted relative paths + sizes) and,
when it changes vs the last compile on this dir, wipe build/ to force a clean
configure. ccache (enabled) refills the objects so the rebuild stays cheap.
No-op on the ephemeral path and the first compile. Fixes both symptoms; will be
superseded by the fully ephemeral per-compile workspace (P1).
A manifest-scoped compile that fails because a header isn't in the manifest
(an undeclared/transitive dependency) now retries once with scan-all, so a
project with an incomplete manifest still compiles instead of regressing.
The response reports manifest_incomplete=true and
manifest_suggested_libraries={header: [candidate lib names]} so the manifest
can be auto-completed (P2.4) or the user prompted to add the missing library.
This de-risks turning on manifest sending (P2.3): an incomplete example/project
manifest can never break a build that worked before.
- compile(): _attempt(allowed) helper; retry scan-all on missing-lib failure.
- _missing_library_headers / _suggest_libraries_for_headers helpers.
- CompileResponse.manifest_incomplete + manifest_suggested_libraries.
P2.0 first cut took the first-alphabetical lib providing a header and then
checked manifest membership. When several installed libs ship the same header
(e.g. DHT118266, DHT_sensor_library, servodht11 all have DHT.h), the stray
first-match got rejected and the header was dropped even though the declared
lib provides it.
_find_manifest_library_for_header: when a manifest is supplied, pick the first
DECLARED library that provides the header. This both selects the right lib and
excludes undeclared ones. No manifest = legacy first-match.
Test strengthened with a stray same-header lib that sorts first.
When a compile supplies a 'libraries' manifest, _resolve_library_components
merges a USER-installed library only if it's declared in that set. A sketch
therefore never picks up an unrelated library from the shared dir (another
user's install, or a same-named clash) — the manifest is the resolution scope.
- _resolve_library_components(allowed_libraries): gate user-lib merges on
manifest membership; match by folder name OR library.properties name=,
normalised (display name vs on-disk folder differ by separators/case).
Core/bundled libs are never gated. None = legacy scan-all (unchanged).
- Threaded through compile() -> _compile_in_dir.
- compile.py: CompileRequest.libraries; folded into the async dedup _job_key
so a different manifest doesn't dedup to a job built with another.
Opt-in: omitting 'libraries' preserves current behaviour exactly.
Regression: test/backend/unit/test_espidf_core_first.py::TestManifestScope
A user library that ships a core-named header (e.g. WiFiEspAT/src/WiFi.h)
could shadow the arduino-esp32 core during ESP-IDF library resolution.
WiFiEspAT shadowing WiFi.h pulled EspAtDrv.cpp into the build, whose
const char OK[]/STATUS[] collide with ESP-IDF's enum STATUS in
rom/ets_sys.h, breaking every ESP32 sketch that #include <WiFi.h>.
_resolve_library_components now:
- skips a header entirely when the arduino-esp32 core provides it
(computed set from cores/ + libraries/, cached), so a user lib can
never shadow WiFi.h/Wire.h/SPI.h/WebServer.h/...
- skips a resolved user lib whose library.properties architectures=
excludes esp32/*.
Regression: test/backend/unit/test_espidf_core_first.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two unrelated polish fixes.
espidf_compiler: headers that resolve to an arduino-esp32 CORE lib
(WebServer, WiFi, …) were correctly skipped from the user-lib merge but
then fell through to a scary "Library for <X> not found — build may
fail" warning — even though the build succeeds because the symbols are
compiled into the core. Now logs an accurate "provided by arduino-esp32
core — already compiled in, not merging". Same treatment for core
headers that aren't standalone lib dirs (Udp.h, IPAddress.h,
WiFiUdp.h, …) via a new _CORE_ESP32_HEADERS allowlist.
SimulatorCanvas: the WiFi badge's "open IoT gateway" click now consults
an optional window.__velxio_iot_gateway_open_gate__ hook before opening
the gateway tab. A private overlay can install it to gate the gateway
behind a paid plan and show an in-place upgrade modal instead of dumping
a 402 page in a new tab. OSS builds have no hook → opens normally. The
check is synchronous so it doesn't trip popup blockers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
When a sketch's external library headers were only referenced from
project headers (e.g. esp32-eyes.ino includes Common.h, Common.h
includes <ESP32Servo.h>), the compile failed with
fatal error: ESP32Servo.h: No such file or directory
because _detect_external_includes was only called on main_content
(the processed .ino). Project .h/.cpp files were never scanned, so
ESP32Servo / DHT / Adafruit_Sensor referenced only transitively
through user code never reached _resolve_library_components and
never landed in user_libs_all/.
Fix: collect ext_headers from main_content PLUS every uploaded
.h/.hpp/.ino/.c/.cpp file before resolving libraries. Lib resolver
already walks transitive includes inside the lib bundle once it's
copied; this just makes sure the first-level set covers user
project headers too.
Repro: open https://velxio.dev/example/robot-desktop-eyes, click
Compile. Before this commit: 13 errors starting at ESP32Servo.h.
After: ext_headers includes ESP32Servo.h on the first pass and the
build proceeds.
The user_libs_all bundler in _resolve_library_components does BFS over the
sketch's external includes, copying each matching Arduino library into
one merged IDF component. Anything not in _BUILTIN_HEADERS is treated as
an external library to resolve, and the lookup just scans
/root/Arduino/libraries/ for a directory whose `src/` (or root) holds a
matching header file.
_BUILTIN_HEADERS only listed C headers (stdint.h, stdio.h, …). The C++
wrappers (cstdint, cstdio, cmath, …) and the STL containers (vector,
complex, string, …) were absent. Result: any library transitively
#including <cstdint> or <vector> caused the bundler to "resolve" the
header against /root/Arduino/libraries/ArduinoSTL/ — an AVR-only
uClibc++ port that ships every C++ stdlib header as plain files. Once
ArduinoSTL was matched the bundler dragged in ALL of it, including
complex.cpp:
template class _UCXXEXPORT complex<float>;
which fails on the ESP-IDF Xtensa toolchain because _UCXXEXPORT isn't
defined in that compile context AND the symbol already exists in the
real libstdc++ pulled in by <complex>. Net effect: every ESP32 sketch
whose deps transitively include a C++ stdlib header (e.g. ESP32Servo
includes <cstdint>) blew up with 66+ errors before the servo example
even reached the link step.
Fix: extend _BUILTIN_HEADERS to cover the full set of C++ stdlib
wrappers and STL headers so the bundler never treats them as installable
libraries. The Xtensa GCC + libstdc++ shipped by ESP-IDF provides them
natively; ArduinoSTL never has any business being part of an ESP32 build.
Verified end-to-end on /example/esp32-servo: compile now succeeds, sketch
boots, moving the potentiometer drives the wokwi-servo angle (Pot=2801 →
Angle=123 deg, servo arm rotates).
Backend:
- api/routes/compile.py accepts board-specific compile options
and dedups in-flight identical requests
- services/espidf_compiler.py expanded ESP-IDF wrapper with the new
options surface (sdkconfig.defaults.in
template added)
- services/arduino_cli.py honour the new options envelope
- services/esp32_lib_bridge.py thread board options through to QEMU
Tests:
- tests/test_compile_request_dedup.py end-to-end dedup behaviour
- tests/test_espidf_options.py covers the new options parsing
Frontend:
- services/compilation.ts client-side mirror — sends the new
options field on every compile request
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A user reported on Discord: "the Velxio Console doesn't update anything,
it just waits until the very end and displays everything in one go".
True for the async compile path — /compile/status only carried `state`
and the final `result`, so the editor's CompilationConsole stayed empty
during the 5-7 minute cold ESP-IDF builds and dumped 1500 lines at once
when the build finished.
This wires live build output through the whole stack.
Backend (espidf_compiler.py)
- New _run_with_streaming() helper. When a progress_callback is provided
it spawns the subprocess via Popen + stdout/stderr drain threads and
invokes the callback line-by-line. When None it falls back to the
existing subprocess.run(capture_output=True) one-shot path so the
unit-test code that doesn't care about live output is unaffected.
- compile() and _compile_in_dir() take an optional ProgressCallback.
- _run_cmake / _run_ninja closures now go through _run_with_streaming
with that callback. cmake configure (~2-5 s) + ninja (~5-300+ s) both
stream now; the ninja output is the one users actually want to watch.
Backend (compile.py)
- _compile_job seeds COMPILE_JOBS[id]['stdout_buffer'] = '' and defines
on_progress_line(line) which appends to it. Buffer capped at 256 KB
(tail kept) so a runaway build can't OOM the FastAPI process.
- The buffer is preserved on both the success and the error path so
late polls still see the log even after state transitions to
done/error.
- /compile/status now returns the buffer as a `stdout` field.
CompileStatusResponse gains the field with default '' so old clients
that don't read it still work.
Frontend (compilation.ts)
- compileCode() takes a 4th argument: optional CompileProgress
callback fired every poll while state ∈ {pending, running}. Carries
the cumulative stdout (caller computes deltas) plus elapsed seconds.
- Surfaces the new `stdout` field of /compile/status and forwards it
to the callback. Errors thrown from the callback are swallowed —
a faulty UI hook must never break the polling loop.
Frontend (EditorToolbar.tsx)
- Both compileCode() call sites (Run and Compile-All) now pass an
onProgress callback. It tracks `lastStreamedLen` per-compile, splits
each new delta on newlines, and appends them as `info`-typed
CompilationLog entries via setCompileLogs. The Compile-All flow
prefixes each line with the board label so multi-board builds stay
readable.
- After the build settles, the existing parseCompileResult call still
runs and appends the structured analysis on top of the live stream
— that's where FAILED-block detection + the `error`-typed entries
that drive the auto-switch-to-errors filter live.
Net effect on the user complaint: cold ESP-IDF builds now show the
ninja [N/1483] progress lines streaming into the console as they
happen, instead of staring at an empty panel for 5-7 minutes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three coordinated fixes that together close the "ESP-IDF compile takes
5-7 min every time" gap and prevent the failure mode where a user clicking
compile multiple times spawns six ninja processes that peel each other
apart on a modest VPS.
What was wrong
- /compile/start generated a fresh uuid4 every call, so 6 clicks = 6
independent builds racing each other. Saw load average 30 on the prod
VPS during a real BMP280 attempt today.
- No concurrency limit anywhere; asyncio.create_task() fired without
gating.
- ccache was wired in last week (PR #149) but reported 18,350 cacheable
calls and **0 hits** because the build dir was a fresh
tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random
/tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags
→ different command line every compile → ccache hash miss every time.
What this PR does
1. Job deduplication (`backend/app/api/routes/compile.py`)
- New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file
names + contents + board. Order-independent.
- New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id.
- `compile_start` checks JOB_BY_KEY before spawning a new task; if a
job for this exact content is already pending or running, returns
the existing job_id (logs `[compile] dedup hit — reusing job <id>`).
- `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY,
keeping the index consistent. Edge case where two jobs share a key
(old finished, new running) is handled — only evict the key entry
if it still points at the purged job.
2. Concurrency control (`backend/app/api/routes/compile.py`)
- `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on
simultaneous compiles.
- `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so
concurrent compiles to the SAME board (sharing the persistent build
dir) serialise. Different boards still run in parallel up to the
semaphore cap.
- `_compile_job` acquires sema → per-target lock → flips state to
`running` → calls `_run_compile`. Pending state now accurately
reflects "queued waiting for resources".
3. Persistent build dir (`backend/app/services/espidf_compiler.py`)
- New `_prepare_persistent_project_dir(idf_target)` materialises
`/var/lib/velxio-build/<target>/project/` from the template on
first use; on subsequent compiles it wipes only `main/` and
`user_libs/` (the per-compile parts) and leaves `build/` alone so
ninja's incremental cache + ccache .o files survive.
- Toolchain version sentinel (`.idf_version`) wipes the whole target
dir if the ESP-IDF or arduino-esp32 version changes — cached
objects from the old toolchain are no longer ABI-compatible.
- `compile()` is now a thin dispatcher: persistent path or fallback
to the legacy `tempfile.TemporaryDirectory()` flow. The actual
build logic was extracted into `_compile_in_dir()` so both paths
share one implementation, no duplication.
- Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back
to the tempfile path without rebuilding the image. Critical for
production safety.
4. ccache normalisation (`Dockerfile.standalone`)
- + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise
absolute paths under that prefix when computing the cache key.
Robustens hits against any future subdir rearrangement.
5. Docker compose (`docker-compose.yml`)
- + named volume `velxio-build:/var/lib/velxio-build` so the persistent
build dir survives `docker compose up -d --build`.
- + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable
without rebuilding).
Expected impact
- Cold first compile per container per target: unchanged (~5-7 min).
- Same sketch re-compiled: ~2-5 s (everything cached).
- Different sketch, same target: ~5-30 s (only user code + new lib steps
rebuild; ESP-IDF base hits cache).
- Different sketch with new libraries: ~30-90 s (new lib component
compiles; rest hits cache).
- Concurrent clicks on same example: 1 build, others poll the same
job_id. No more six-ninja meltdown.
Tests
- `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability +
variance and `_purge_expired_jobs` consistency (including the
"two jobs share a key" edge case).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cold first compile per container is unchanged (cache empty). Subsequent
compiles drop from ~5-7 minutes to ~30-60 seconds because every ESP-IDF
base object (FreeRTOS, lwIP, esp_wifi, libsodium, soc, hal, …) hits the
cache. The user's BMP280 example, which hangs on cold compile, completes
near-instantly on the second attempt.
Why a transparent cache is safe: ccache hashes the preprocessed source +
flags + compiler. A cache hit only happens when the input is byte-for-byte
identical to a prior compile. Different sketches with different libraries
still get correct cache misses; there is no path where one project's
output contaminates another.
Changes
- Dockerfile.standalone: install ccache, set CCACHE_DIR=/var/cache/ccache,
IDF_CCACHE_ENABLE=1, configure 2 GB cap with compression. Compression
(level 6) cuts cache disk usage by ~40% with negligible CPU overhead.
- docker-compose.yml: named volume `ccache:/var/cache/ccache` so the
cache survives `docker compose up -d --build` (without it, every image
rebuild discards the cache).
- backend/app/services/espidf_compiler.py: pass `-DCCACHE_ENABLE=1` to
cmake when IDF_CCACHE_ENABLE is truthy. ESP-IDF's project.cmake
(`set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)` on line 374)
is what actually wires ccache in; without the cmake -D flag the env
var alone has no effect because we don't go through idf.py.
Escape hatch: set IDF_CCACHE_ENABLE=0 in compose env to disable without
rebuilding the image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ESP32 BMP280 example compile was timing out at 98% (1473/1483 build
steps), failing with the unhelpful "ESP-IDF build timed out (300s)"
message even though every individual step was healthy.
Cold ESP-IDF builds that pull in external Arduino libraries — Adafruit
BMP280 + Adafruit BusIO + Adafruit Unified Sensor on top of the base
arduino-esp32 component tree — routinely produce ~1480 build objects.
On modest VPS hardware this takes 5-7 minutes the first time. Ninja's
incremental cache makes subsequent compiles seconds, but the first one
needs more headroom.
Constant lifted to NINJA_TIMEOUT_S so the value used in the timeout
matches the value reported in the error message — the previous code
hard-coded "300s" in two places that were free to drift apart.
Repro before: open the example "ESP32 — BMP280 Barometric Pressure"
on velxio.dev/editor on a clean container, click compile → fails after
5 minutes with timeout. After: completes in ~6 minutes on the first
run, ~5 seconds on subsequent runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A user reported the LEDC PWM RGB example failing to compile. The sketch
calls ledcAttach(pin, freq, resolution) — the one-shot API added in
arduino-esp32 3.x. Our toolchain image pins arduino-esp32 to 2.0.17
(matched to ESP-IDF 4.4.7 + the lcgamboa QEMU ROM), where the API is
the older two-step ledcSetup + ledcAttachPin pair. Sketches written
against 3.x docs hit "ledcAttach was not declared in this scope".
Bumping arduino-esp32 to 3.x means moving to ESP-IDF 5.x, which may
break our QEMU fork. Cheaper fix: ship a compat shim header in the
ESP-IDF project template that defines ledcAttach + ledcAttachChannel
in terms of the 2.x API, gated on `!defined(ledcAttach)` so it
disappears the day we bump.
espidf_compiler.py now injects #include "velxio_compat.h" right after
Arduino.h whether the user explicitly included Arduino.h or we
prepended it ourselves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issue #101 reproducer: an ESP32 sketch that pulls in Adafruit_SSD1306 +
Adafruit_GFX produced a "No response from server. Is the backend
running on port 8001?" error in the browser. The compile actually
succeeded backend-side, but the JSON response carrying the firmware
was ~5.5 MB of base64 — the ESP-IDF compiler builds a full 4 MB merged
flash image (mostly 0xFF padding), encodes it whole, and ships it. In
prod that response goes through nginx + Cloudflare, which buffer-fail
or RST the connection on payloads that big — axios then lands in the
"no response" branch with no HTTP status to surface.
Fix: trim the trailing 0xFF padding before serializing, re-pad to a
valid QEMU flash size (2/4/8/16 MB) just before mtd attach. Lossless:
bytes after `last_used` in the merge are 0xFF by construction, so
trim → pad reproduces the original image byte-for-byte.
Numbers from the reproducer (Adafruit_SSD1306 + Adafruit_GFX,
esp32:esp32:esp32 board):
before: ~5.5 MB JSON response
after: 539 KB JSON response (10× smaller)
backend/app/services/espidf_compiler.py
_merge_flash_image now tracks `last_used` across the three placed
sections (bootloader / partitions / app) and writes only
flash[:last_used] to merged_flash.bin.
backend/app/services/esp32_flash_image.py (new)
Shared `pad_to_flash_size(bytes) -> bytes` helper. Rounds up to the
next valid QEMU flash size with a 4 MB minimum, matches the
frontend's existing padToFlashSize logic in Esp32MicroPythonLoader.
Raises ValueError on >16 MB inputs (would indicate a broken upstream
merge, not anything user-recoverable).
backend/app/services/esp32_lib_bridge.py
backend/app/services/esp32_worker.py
Both QEMU consumer paths (in-process and subprocess) call
pad_to_flash_size right after base64.b64decode, before writing the
tmp .bin that QEMU attaches with `-drive if=mtd,format=raw`.
Verified: smoke test confirms trim → pad → original is byte-exact.
Edge cases covered: small payloads pad up to the 4 MB minimum;
firmwares >16 MB are rejected loudly.
Closes#101
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The _should_include filter inside _merge_arduino_libs_to_component
rejected every subdirectory of a library except 'utility/' when the
library lacked a src/ layout. That blocked legitimate header dirs like
Adafruit_GFX_Library/Fonts/, breaking compiles that use any GxEPD2
example with a custom font (#include <Fonts/FreeMonoBold12pt7b.h>).
The earlier filter at the top of the function already excludes
docs/examples/tests/etc. via excluded_dirs, so anything that survives
that check is presumed to be buildable source. Letting all remaining
subdirs through restores Fonts/, gfxfont/, and similar conventional
auxiliary header directories that Adafruit-style libs rely on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The transitive-include scan in _merge_arduino_libs_to_component used
glob('*.h') on the component dir, which only finds headers at the root.
Libraries with a src/ layout (GxEPD2, ArduinoJson, most modern Arduino
libs) keep their headers under src/<...>, so the scan saw zero headers
and never queued their transitive deps.
Symptom: compiling a sketch that includes GxEPD2_3C.h failed with
'Adafruit_GFX.h: No such file or directory' even though Adafruit_GFX
was installed via the Library Manager — because the BFS never reached
its header from inside GxEPD2_GFX.h.
Switching to rglob('*.h') walks the full directory tree and lets the
BFS pick up Adafruit_GFX, Adafruit_BusIO, and any other transitive
dependency that lives under src/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The directory grew well beyond Wokwi-only contents: it now hosts
lcgamboa's QEMU fork (qemu-lcgamboa), Espressif's esp32-camera, the
ngspice WASM build, fritzing-parts, picowi, an alternative QEMU
(qemu-esp32), the 100_Days_100_IoT_Projects examples repo, and
Wokwi's own avr8js/rp2040js/wokwi-elements/wokwi-features/wokwi-boards.
"wokwi-libs" was misleading — half the contents have nothing to do
with Wokwi. "third-party/" is the standard convention for vendored
external dependencies.
Mechanical changes:
Path rename:
wokwi-libs/ → third-party/
update-wokwi-libs.bat → update-third-party.bat
docs/WOKWI_LIBS.md → docs/THIRD_PARTY.md
Submodule reconfiguration:
.gitmodules — 4 path= and section names updated
.git/modules/wokwi-libs/ → .git/modules/third-party/
each submodule's .git file rewired to ../../.git/modules/third-party/<name>
Reference updates (~80 files): vite.config.ts aliases, Dockerfile
COPY paths, GH Actions workflow steps, build_qemu_*.sh, all
docs/* and test/*/autosearch/* entries that mention the path,
package-lock.json file: dependencies, .gitignore patterns,
sitemap.xml + index.html SEO blurbs, scripts/generate-component-*,
.dockerignore, .idea/vcs.xml. Bulk replaced both `wokwi-libs/`
(path) and bare `wokwi-libs` (textual mentions in docs/comments).
Verified:
- npx tsc -b --noEmit produces no new errors related to these paths
- vite.config.ts aliases now point at ../third-party/avr8js etc.
- All 4 git submodules (avr8js, rp2040js, wokwi-elements,
wokwi-features) are linked under third-party/ with their
worktrees re-populated and config files referencing the new path
- `grep -r wokwi-libs` returns zero hits outside node_modules,
.vite, frontend/dist, third-party/ (upstream submodule contents),
*.pyc caches, and *.dll.pre-camera rollback binaries
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_create_idf_component was fixed but NOT _merge_arduino_libs_to_component.
The latter was still flattening library files, causing ArduinoJson compilation to fail
with 'src/ArduinoJson.h: No such file or directory'.
Changes:
- Replace flat copy with directory-structure-preserving copy
- Add exclusion logic for non-buildable directories (examples, tests, docs)
- Generate INCLUDE_DIRS from actual directory structure
- Track files by relative path to prevent name collisions
This ensures libraries with src/ layouts (ArduinoJson, etc.) compile correctly.
- Maintain original library layout (src/, utility/) instead of flattening files
- Add exclusion logic for non-buildable directories (examples, tests, docs, CI)
- Dynamically generate INCLUDE_DIRS from actual directory structure
- Add validation to ensure buildable source files exist before proceeding
- Support both flat and src-based library layouts
- Fix path separator normalization for cross-platform compatibility
This improves compatibility with complex Arduino libraries that rely on
specific directory structures and relative includes.
refactor: Update wiring examples for E32 OLED integration; correct pin mappings for VCC, GND, DATA, and CLK
test: Improve unit tests for ESPIDFCompiler; add scenarios for library resolution and CMake patching
chore: Mark subproject commits as dirty for wokwi-libs
Two fixes for ESP32 WiFi not connecting in production:
1. espidf_compiler.py: Change WiFi normalization from 'Velxio-GUEST' on
channel 6 to 'Espressif' on channel 5. The lcgamboa QEMU binary
downloaded from GitHub Releases only contains the original three APs:
PICSimLabWifi (ch1), Espressif (ch5), MasseyWifi (ch10). Channel 6
had no matching AP, so the beacon timer's channel-match condition never
fired → firmware scanned forever and never connected.
2. esp32_worker.py: Redirect fd 1 to /dev/null before loading QEMU so
raw UART bytes from QEMU's -nographic mux don't corrupt the JSON
event pipe. The real pipe fd is saved and sys.stdout is rebound so
_emit() continues to work. This also prevents stdout pipe back-pressure
from stalling qemu_main_loop() (and thus REALTIME timers).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Frontend WiFi detection via file-content scanning was unreliable because
fileGroups[board.activeFileGroupId] could be an empty array (not null),
bypassing the ?? fallback to editorState.files.
Fix: the ESP-IDF compiler now returns has_wifi:bool in its compile response.
The frontend stores this on the BoardInstance and uses it in startBoard()
instead of scanning file contents. The file-content scan is kept as a
fallback for boards that haven't been compiled in this session.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_build_env() on Linux only set IDF_TOOLS_PATH but never added the tool
binary directories to PATH, so cmake could not find riscv32-esp-elf-g++
when compiling for ESP32-C3. Also improve ninja failure logging to show
stdout (where build errors actually appear) instead of empty stderr.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Dockerfile: download pre-built .so + ROM from velxio public release
instead of building from private qemu-lcgamboa source
- espidf_compiler: normalize any WiFi SSID → "Velxio-GUEST" for QEMU
compatibility (channel 6, open auth)
- docker-compose.yml: unified dev/prod using Dockerfile.standalone
- .dockerignore: exclude qemu-lcgamboa source from Docker context
- .gitignore: ignore prebuilt/ binaries, keep .gitkeep
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace arduino-cli with ESP-IDF 4.4.7 for ESP32 compilation — Arduino-compiled
firmware crashes in QEMU (9-28 reboots) while ESP-IDF boots cleanly (0 reboots).
The new espidf_compiler translates Arduino WiFi/WebServer sketches to native
ESP-IDF C code, compiles with cmake+ninja, and merges into 4MB flash images.
Key changes:
- ESP-IDF compiler: translates WiFi.begin/WebServer to esp_wifi/esp_http_server
- ESP-IDF project template with QEMU-optimized sdkconfig (DIO, 40MHz, no WDT)
- WiFi status parser for ESP-IDF serial logs (wifi_status, ble_status events)
- IoT Gateway HTTP reverse proxy for ESP32 web servers
- WiFi/BLE auto-detection from sketch content + visual status icons
- Static IP 192.168.4.15 matching slirp DHCP first-client range
- Docker: new espidf-builder stage with ESP-IDF 4.4.7 toolchain
- 157 tests covering WiFi/BLE for both ESP32 (Xtensa) and ESP32-C3 (RISC-V)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
David Montero
ZhadowValker