c2fe1af250
Three coordinated fixes that together close the "ESP-IDF compile takes 5-7 min every time" gap and prevent the failure mode where a user clicking compile multiple times spawns six ninja processes that peel each other apart on a modest VPS. What was wrong - /compile/start generated a fresh uuid4 every call, so 6 clicks = 6 independent builds racing each other. Saw load average 30 on the prod VPS during a real BMP280 attempt today. - No concurrency limit anywhere; asyncio.create_task() fired without gating. - ccache was wired in last week (PR #149) but reported 18,350 cacheable calls and **0 hits** because the build dir was a fresh tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random /tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags → different command line every compile → ccache hash miss every time. What this PR does 1. Job deduplication (`backend/app/api/routes/compile.py`) - New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file names + contents + board. Order-independent. - New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id. - `compile_start` checks JOB_BY_KEY before spawning a new task; if a job for this exact content is already pending or running, returns the existing job_id (logs `[compile] dedup hit — reusing job <id>`). - `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY, keeping the index consistent. Edge case where two jobs share a key (old finished, new running) is handled — only evict the key entry if it still points at the purged job. 2. Concurrency control (`backend/app/api/routes/compile.py`) - `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on simultaneous compiles. - `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so concurrent compiles to the SAME board (sharing the persistent build dir) serialise. Different boards still run in parallel up to the semaphore cap. - `_compile_job` acquires sema → per-target lock → flips state to `running` → calls `_run_compile`. Pending state now accurately reflects "queued waiting for resources". 3. Persistent build dir (`backend/app/services/espidf_compiler.py`) - New `_prepare_persistent_project_dir(idf_target)` materialises `/var/lib/velxio-build/<target>/project/` from the template on first use; on subsequent compiles it wipes only `main/` and `user_libs/` (the per-compile parts) and leaves `build/` alone so ninja's incremental cache + ccache .o files survive. - Toolchain version sentinel (`.idf_version`) wipes the whole target dir if the ESP-IDF or arduino-esp32 version changes — cached objects from the old toolchain are no longer ABI-compatible. - `compile()` is now a thin dispatcher: persistent path or fallback to the legacy `tempfile.TemporaryDirectory()` flow. The actual build logic was extracted into `_compile_in_dir()` so both paths share one implementation, no duplication. - Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back to the tempfile path without rebuilding the image. Critical for production safety. 4. ccache normalisation (`Dockerfile.standalone`) - + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise absolute paths under that prefix when computing the cache key. Robustens hits against any future subdir rearrangement. 5. Docker compose (`docker-compose.yml`) - + named volume `velxio-build:/var/lib/velxio-build` so the persistent build dir survives `docker compose up -d --build`. - + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable without rebuilding). Expected impact - Cold first compile per container per target: unchanged (~5-7 min). - Same sketch re-compiled: ~2-5 s (everything cached). - Different sketch, same target: ~5-30 s (only user code + new lib steps rebuild; ESP-IDF base hits cache). - Different sketch with new libraries: ~30-90 s (new lib component compiles; rest hits cache). - Concurrent clicks on same example: 1 build, others poll the same job_id. No more six-ninja meltdown. Tests - `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability + variance and `_purge_expired_jobs` consistency (including the "two jobs share a key" edge case). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| autosearch | ||
| backend | ||
| esp32 | ||
| esp32-emulator | ||
| esp32_cam | ||
| multi_board | ||
| multi_board_esp32 | ||
| pi_arduino_serial | ||
| test-esp32-cam | ||
| test-esp32-p4 | ||
| test_100_days | ||
| test_Raspberry_Pi_Pico_W | ||
| test_chip_backend_runtime | ||
| test_circuit | ||
| test_custom_chips | ||
| test_custom_chips_boards | ||
| test_epaper | ||
| test_intel | ||
| docker-errors-2026-04-19.md | ||