A user reported on Discord that an ESP32 Blink compile on a fresh
`docker run -d ghcr.io/.../velxio:master` (no -v flags) takes 8-9
minutes — every single time. Same hardware that the local project
checkout flies through in 5-30 seconds with the new persistent build
dir + ccache pipeline.
Root cause: `docker run` without `-v` mounts gets nothing persistent.
ccache + persistent build dir live in /var/cache/ccache and
/var/lib/velxio-build, both wiped on every `docker rm` (which the
user explicitly did when troubleshooting). docker-compose users get
the volumes via the compose file; standalone users got nothing
because the Dockerfile didn't declare them.
This PR closes that gap.
Dockerfile.standalone
- VOLUME ["/app/data", "/root/.arduino15", "/root/Arduino",
"/var/cache/ccache", "/var/lib/velxio-build"]
Anonymous volumes are now created automatically when the user runs
the image without -v. They survive `docker stop`/`docker start`/
`docker rm` (only `docker rm -v` or `docker volume prune` removes
them). Users can still pass `-v` for named volumes — explicit
mounts always win over the VOLUME directive.
- Replace `ccache --set-config max_size 8G` (RUN, written to
/var/cache/ccache/ccache.conf which the volume mount masks at
runtime) with `ENV CCACHE_MAXSIZE=8G` etc. — env vars override any
conf-file value on every ccache invocation, so the 8 GB cap
actually applies at runtime regardless of what's in the volume.
README.md
- Update both the quick-start docker run and the detailed self-host
section to include all five volumes.
- Add a note explaining what each volume is for and that without them,
cold compile times are 5-7 min vs the 5-30 s warm path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous bump to 8G (PR #153) didn't take effect on prod. Verified
post-deploy:
Cache size (GB): 2.0 / 2.0 (99.95%)
Cause: $CCACHE_DIR (/var/cache/ccache) is a named docker volume. Any
config we wrote into it via `ccache --set-config max_size 8G` during
the RUN step gets masked at runtime by the mount of the existing
volume — which still contained the original 2 GB config from when the
cache was first populated.
Fix: set CCACHE_MAXSIZE / CCACHE_COMPRESS / CCACHE_COMPRESSLEVEL as
ENV in the Dockerfile. ccache reads those at runtime and they
override any conf-file value, including whatever stale config is
sitting in the volume.
After this lands and the submodule bumps, `ccache -s` should report
the new 8 GB max immediately on container start without needing to
wipe or re-init the cache volume.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Empirical confirmation from a smoke test on prod after PR #152
(persistent build dir + dedup) landed:
Cache size (GB): 2.0 / 2.0 (99.96%)
Hits: 1 / 86886 ( 0.00%)
Misses: 86885 / 86886 (100.0%)
The cap was set to 2G when ccache was first wired in (PR #149) under
the assumption that base ESP-IDF would fit. With arduino-esp32 +
external Arduino libraries (Adafruit BMP280 + BusIO + Unified_Sensor +
GFX + …) the cacheable object set comfortably exceeds that — we
observed 86k misses while the cache was permanently at 99.96% full,
meaning entries get evicted before subsequent compiles can hit them.
Disk usage on the prod VPS at /var/cache/ccache will rise from ~2 GB
to ~6-8 GB. The host has plenty of headroom and the named volume sits
on the same partition as the rest of /var. Compression at level 6 is
already on, so the disk impact is the on-disk size after compression.
The 13-second warm compile we measured today already proves the bulk
of the perf win came from ninja's incremental cache + persistent build
dir; this is the cherry on top that lets ccache start contributing
across container restarts (when ninja's in-build-dir cache is fresh
but ccache should still hit at the C/C++ object level).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three coordinated fixes that together close the "ESP-IDF compile takes
5-7 min every time" gap and prevent the failure mode where a user clicking
compile multiple times spawns six ninja processes that peel each other
apart on a modest VPS.
What was wrong
- /compile/start generated a fresh uuid4 every call, so 6 clicks = 6
independent builds racing each other. Saw load average 30 on the prod
VPS during a real BMP280 attempt today.
- No concurrency limit anywhere; asyncio.create_task() fired without
gating.
- ccache was wired in last week (PR #149) but reported 18,350 cacheable
calls and **0 hits** because the build dir was a fresh
tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random
/tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags
→ different command line every compile → ccache hash miss every time.
What this PR does
1. Job deduplication (`backend/app/api/routes/compile.py`)
- New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file
names + contents + board. Order-independent.
- New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id.
- `compile_start` checks JOB_BY_KEY before spawning a new task; if a
job for this exact content is already pending or running, returns
the existing job_id (logs `[compile] dedup hit — reusing job <id>`).
- `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY,
keeping the index consistent. Edge case where two jobs share a key
(old finished, new running) is handled — only evict the key entry
if it still points at the purged job.
2. Concurrency control (`backend/app/api/routes/compile.py`)
- `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on
simultaneous compiles.
- `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so
concurrent compiles to the SAME board (sharing the persistent build
dir) serialise. Different boards still run in parallel up to the
semaphore cap.
- `_compile_job` acquires sema → per-target lock → flips state to
`running` → calls `_run_compile`. Pending state now accurately
reflects "queued waiting for resources".
3. Persistent build dir (`backend/app/services/espidf_compiler.py`)
- New `_prepare_persistent_project_dir(idf_target)` materialises
`/var/lib/velxio-build/<target>/project/` from the template on
first use; on subsequent compiles it wipes only `main/` and
`user_libs/` (the per-compile parts) and leaves `build/` alone so
ninja's incremental cache + ccache .o files survive.
- Toolchain version sentinel (`.idf_version`) wipes the whole target
dir if the ESP-IDF or arduino-esp32 version changes — cached
objects from the old toolchain are no longer ABI-compatible.
- `compile()` is now a thin dispatcher: persistent path or fallback
to the legacy `tempfile.TemporaryDirectory()` flow. The actual
build logic was extracted into `_compile_in_dir()` so both paths
share one implementation, no duplication.
- Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back
to the tempfile path without rebuilding the image. Critical for
production safety.
4. ccache normalisation (`Dockerfile.standalone`)
- + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise
absolute paths under that prefix when computing the cache key.
Robustens hits against any future subdir rearrangement.
5. Docker compose (`docker-compose.yml`)
- + named volume `velxio-build:/var/lib/velxio-build` so the persistent
build dir survives `docker compose up -d --build`.
- + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable
without rebuilding).
Expected impact
- Cold first compile per container per target: unchanged (~5-7 min).
- Same sketch re-compiled: ~2-5 s (everything cached).
- Different sketch, same target: ~5-30 s (only user code + new lib steps
rebuild; ESP-IDF base hits cache).
- Different sketch with new libraries: ~30-90 s (new lib component
compiles; rest hits cache).
- Concurrent clicks on same example: 1 build, others poll the same
job_id. No more six-ninja meltdown.
Tests
- `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability +
variance and `_purge_expired_jobs` consistency (including the
"two jobs share a key" edge case).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cold first compile per container is unchanged (cache empty). Subsequent
compiles drop from ~5-7 minutes to ~30-60 seconds because every ESP-IDF
base object (FreeRTOS, lwIP, esp_wifi, libsodium, soc, hal, …) hits the
cache. The user's BMP280 example, which hangs on cold compile, completes
near-instantly on the second attempt.
Why a transparent cache is safe: ccache hashes the preprocessed source +
flags + compiler. A cache hit only happens when the input is byte-for-byte
identical to a prior compile. Different sketches with different libraries
still get correct cache misses; there is no path where one project's
output contaminates another.
Changes
- Dockerfile.standalone: install ccache, set CCACHE_DIR=/var/cache/ccache,
IDF_CCACHE_ENABLE=1, configure 2 GB cap with compression. Compression
(level 6) cuts cache disk usage by ~40% with negligible CPU overhead.
- docker-compose.yml: named volume `ccache:/var/cache/ccache` so the
cache survives `docker compose up -d --build` (without it, every image
rebuild discards the cache).
- backend/app/services/espidf_compiler.py: pass `-DCCACHE_ENABLE=1` to
cmake when IDF_CCACHE_ENABLE is truthy. ESP-IDF's project.cmake
(`set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)` on line 374)
is what actually wires ccache in; without the cmake -D flag the env
var alone has no effect because we don't go through idf.py.
Escape hatch: set IDF_CCACHE_ENABLE=0 in compose env to disable without
rebuilding the image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A lock file pins platform-specific native binaries — Rollup, esbuild, swc.
A lock generated on Windows brings @rollup/rollup-win32-x64-msvc but no
Linux variant; a lock generated on Linux does the inverse. The Docker
build kept blowing up with MODULE_NOT_FOUND on rollup/dist/native.js
whenever the lock came from a contributor's non-Linux machine.
Trade-off: we lose npm's transitive-version pinning. Mitigated by:
- package.json caret ranges keep majors stable
- Docker image is rebuilt + retagged per release, so a deployed image
has a frozen dep set regardless of the lock
- Production uses a pinned upstream commit via velxio-prod's submodule,
not lock-driven repro
- Dependabot still flags vulnerable transitives via package.json scans
Changes:
- .gitignore: ignore package-lock.json everywhere
- .dockerignore: same (defense-in-depth — never enter build context)
- Dockerfile.standalone: keep `rm -f package-lock.json` as a safety net
for `docker build` runs from trees with a local lock
- frontend-tests.yml: `npm ci` → `npm install` (npm ci requires a lock)
- Delete the two committed locks (frontend/ + root). The test/* and
vscode-extension/* locks are left as-is — internal tooling, separate
install paths, not in the Docker build.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lock files generated on Windows/macOS pin platform-specific Rollup native
binaries (e.g. @rollup/rollup-win32-x64-msvc) and won't bring in the Linux
ones the Docker image needs. Symptom: `MODULE_NOT_FOUND` on
`rollup/dist/native.js` during `npm run build:docker`. See npm/cli#4828.
Removing the lock inside the build forces a fresh, Linux-native dep
resolution. Side effect: a tiny loss of cross-build version pinning, which
is acceptable here — `npm install` honours the version ranges in
package.json so semver-compatible patches at most slip in.
The proper long-term fix is to regenerate package-lock.json on Linux and
commit that. Until then this rm guards every Docker build.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The folder only holds the in-container nginx + entrypoint that the
standalone Dockerfile copies. Calling it "deploy" implied host-level
production glue, which now lives in github.com/velxio/velxio-prod.
"docker/" makes the build-time vs deploy-time split obvious.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves several install pain points reported by users (#108, #120) and
removes the obligatory upstream-clone step that confused contributors and
slowed down every Docker build.
Install fixes:
- nginx: server_name → catch-all default_server, drop Debian's stock site
so reverse-proxied users no longer get the "Welcome to nginx" page.
- entrypoint: auto-generate SECRET_KEY at first boot, persisted under
data/.secret_key. backend/.env is now optional in docker-compose.yml.
- backend: add greenlet>=3.0.0 (SQLAlchemy async dep that was missing on
some Python builds — caused uvicorn startup failures on WSL).
Wokwi libs come from npm:
- @wokwi/elements 1.9.2, avr8js 0.21.0, rp2040js 1.3.2 are pinned in
frontend/package.json. Vite aliases removed.
- Dockerfile.standalone no longer clones avr8js / rp2040js / wokwi-elements
/ wokwi-boards. Frontend stage is just COPY + npm install + build:docker.
- Board SVGs vendored under frontend/public/boards/ (10 deduped against
existing files, 2 truly new). third-party/wokwi-* clones become reference-
only credits — generate-component-metadata.ts skips gracefully when absent.
Production config split out:
- docker-compose.prod.yml, deploy/nginx.prod.conf, nginx-host-velxio*.conf,
update-third-party.bat removed. Production deployment lives in its own
repo: https://github.com/velxio/velxio-prod (host nginx + HTTPS + backups
+ pinned upstream commit).
Verified locally: 1161 frontend tests pass, build:docker completes clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The directory grew well beyond Wokwi-only contents: it now hosts
lcgamboa's QEMU fork (qemu-lcgamboa), Espressif's esp32-camera, the
ngspice WASM build, fritzing-parts, picowi, an alternative QEMU
(qemu-esp32), the 100_Days_100_IoT_Projects examples repo, and
Wokwi's own avr8js/rp2040js/wokwi-elements/wokwi-features/wokwi-boards.
"wokwi-libs" was misleading — half the contents have nothing to do
with Wokwi. "third-party/" is the standard convention for vendored
external dependencies.
Mechanical changes:
Path rename:
wokwi-libs/ → third-party/
update-wokwi-libs.bat → update-third-party.bat
docs/WOKWI_LIBS.md → docs/THIRD_PARTY.md
Submodule reconfiguration:
.gitmodules — 4 path= and section names updated
.git/modules/wokwi-libs/ → .git/modules/third-party/
each submodule's .git file rewired to ../../.git/modules/third-party/<name>
Reference updates (~80 files): vite.config.ts aliases, Dockerfile
COPY paths, GH Actions workflow steps, build_qemu_*.sh, all
docs/* and test/*/autosearch/* entries that mention the path,
package-lock.json file: dependencies, .gitignore patterns,
sitemap.xml + index.html SEO blurbs, scripts/generate-component-*,
.dockerignore, .idea/vcs.xml. Bulk replaced both `wokwi-libs/`
(path) and bare `wokwi-libs` (textual mentions in docs/comments).
Verified:
- npx tsc -b --noEmit produces no new errors related to these paths
- vite.config.ts aliases now point at ../third-party/avr8js etc.
- All 4 git submodules (avr8js, rp2040js, wokwi-elements,
wokwi-features) are linked under third-party/ with their
worktrees re-populated and config files referencing the new path
- `grep -r wokwi-libs` returns zero hits outside node_modules,
.vite, frontend/dist, third-party/ (upstream submodule contents),
*.pyc caches, and *.dll.pre-camera rollback binaries
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The project save/load pipeline only persisted a single `board_type`, so
multi-board workspaces silently lost every board except the active one
on save, and wires referencing the dropped boards' IDs orphaned to the
canvas corner on reload. An audit of the production backup found 74/306
projects (24%) with at least one orphaned wire and 174/301 non-trivial
projects whose code was still the default Blink template — strong signal
that users save once and never re-save.
Backend
- Add `boards_json` column on `projects` with idempotent ALTER TABLE in
the lifespan migration list.
- New `FileGroup` schema + `file_groups` array on
ProjectCreate/Update/Response. Legacy `files`/`code` kept for back-compat.
- `project_files.py` now uses `{pid}/{groupId}/{filename}` subdirs via
`read_groups`/`write_groups`. Legacy flat layouts are auto-promoted on
read; legacy single-list `files` only updates the active group, leaving
other boards' files intact.
- `_persist_files_from_body` honors file_groups → files → code priority.
Frontend
- `useSimulatorStore.addBoard` accepts an optional `explicitId` so
saved board IDs can be restored verbatim (wires reference IDs literally).
- New `loadProjectState({boards, fileGroups, components, wires,
activeBoardId})` action: tears down current boards, recreates from the
payload, restores file groups atomically, recalculates wire positions
on the next frame, and refreshes the Interconnect.
- `useEditorStore.replaceFileGroups` for atomic multi-group restore.
- `SaveProjectModal` and `ProjectByIdPage`/`ProjectPage` now go through
`buildSavePayload` / `buildLoadPayload` (handles pre-backfill projects
by synthesising a default board from `board_type`).
Auto-save (#useAutoSaveProject hook)
- 2.5s debounced silent PUT triggered ONLY when an authenticated user
has a `currentProject` with a UUID. State hash detects real changes
vs. UI-only churn; baseline is reset on project load so the just-loaded
state isn't immediately re-saved.
- `beforeunload` flush via `fetch keepalive: true` (supports PUT +
credentials, survives unload).
- Compact status indicator in `AppHeader` (idle/dirty/saving/saved/error).
Backfill script (one-off, idempotent)
- `backend/scripts/backfill_boards_2026_05.py` populates `boards_json`
for legacy projects. Heuristic per project, based on which board IDs
the wires reference:
Case A — wires only ref 'arduino-uno' but board_type ≠ uno:
rename id→board_type and rewrite wire endpoints.
Case B — single-board normal: keep verbatim.
Case C — multi-board: recreate one board per distinct ref, infer
kind by stripping trailing -N suffix.
Also moves any flat files into the active board's group subdir.
Stdlib-only, runs from host or `docker exec`.
Docker
- `Dockerfile.standalone` now copies `backend/scripts/` into the image
so the backfill is callable via `docker exec velxio-app python
/app/scripts/backfill_boards_2026_05.py --apply`.
Verified locally on the restored production backup (363 projects):
33 Case A, 316 Case B, 14 Case C, 135 wire endpoints renamed, 0 orphans.
Re-running the script after apply skips all 363 (idempotent).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Docker multi-arch:
- Dockerfile downloads arch-specific QEMU .so via TARGETARCH
- docker-publish.yml adds setup-qemu-action and platforms: linux/amd64,linux/arm64
- qemu-lcgamboa submodule updated (matrix build for both architectures)
LED fix:
- LEDs now require cathode wired to GND (or LOW GPIO) to light up
- Previously LEDs turned on with anode HIGH regardless of cathode connection
- Updated tests to verify anode+cathode behavior
- Dockerfile: download pre-built .so + ROM from velxio public release
instead of building from private qemu-lcgamboa source
- espidf_compiler: normalize any WiFi SSID → "Velxio-GUEST" for QEMU
compatibility (channel 6, open auth)
- docker-compose.yml: unified dev/prod using Dockerfile.standalone
- .dockerignore: exclude qemu-lcgamboa source from Docker context
- .gitignore: ignore prebuilt/ binaries, keep .gitkeep
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ESP-IDF 4.4.x requires specific package versions (reedsolo<=1.5.4,
bitstring<4, pyparsing<2.4.0, kconfiglib==13.7.1, etc.) that conflict
with newer defaults. Installing requirements.txt after copying ESP-IDF
ensures the correct versions are used and cmake configure succeeds.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- git is required by ESP-IDF cmake to resolve submodules and version info
- packaging Python package is required by ESP-IDF Python dependency checks
during cmake configure step
Without these, ESP32 compilation fails at cmake configure with:
"Some Python dependencies must be installed"
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- cmake and ninja are required by ESP-IDF compilation pipeline but were
missing from the final stage, causing all ESP32 compilations to fail
- libusb-1.0-0 fixes the openocd-esp32 shared library warning on startup
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace arduino-cli with ESP-IDF 4.4.7 for ESP32 compilation — Arduino-compiled
firmware crashes in QEMU (9-28 reboots) while ESP-IDF boots cleanly (0 reboots).
The new espidf_compiler translates Arduino WiFi/WebServer sketches to native
ESP-IDF C code, compiles with cmake+ninja, and merges into 4MB flash images.
Key changes:
- ESP-IDF compiler: translates WiFi.begin/WebServer to esp_wifi/esp_http_server
- ESP-IDF project template with QEMU-optimized sdkconfig (DIO, 40MHz, no WDT)
- WiFi status parser for ESP-IDF serial logs (wifi_status, ble_status events)
- IoT Gateway HTTP reverse proxy for ESP32 web servers
- WiFi/BLE auto-detection from sketch content + visual status icons
- Static IP 192.168.4.15 matching slirp DHCP first-client range
- Docker: new espidf-builder stage with ESP-IDF 4.4.7 toolchain
- 157 tests covering WiFi/BLE for both ESP32 (Xtensa) and ESP32-C3 (RISC-V)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Updated components-metadata.json with new generated timestamp.
- Refactored Esp32C3Simulator.ts to remove unnecessary debug variables and logging, and added support for additional ROM functions.
- Modified useSimulatorStore.ts to clarify bridge usage for ESP32 boards.
- Updated submodules for QEMU and other libraries to indicate dirty state.
- Added test_esp32c3_emulation.py for end-to-end testing of ESP32-C3 emulation, including compilation, flash image merging, and GPIO event checking.
Mount ./data:/app/data in both compose files and set DATABASE_URL to
the absolute path /app/data/velxio.db so the database survives container
restarts and updates. Create /app/data in Dockerfile.standalone.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add LoginPage and RegisterPage for user authentication.
- Create UserProfilePage to display user projects.
- Implement ProjectPage for viewing and editing individual projects.
- Introduce authService for handling user login, registration, and session management.
- Add projectService for managing project data retrieval and manipulation.
- Enhance EditorPage with file management capabilities and save prompts.
- Introduce Zustand stores for managing authentication, editor state, and project state.
- Add reserved usernames utility to prevent certain usernames during registration.
- Update compilation service to handle multiple files for Arduino sketches.
davidmonterocrespo24