Commit Graph

24 Commits

Author SHA1 Message Date
David Montero Crespo c8ddfdddba fix(espidf): las cabeceras incluidas con comillas tambien resuelven libreria
Arduino trata igual #include "Lib.h" que #include <Lib.h> para librerias, y
los ejemplos de los propios fabricantes usan la forma con comillas: los de
M5Stack para el Cardputer abren con #include "M5Cardputer.h". Al escanear solo
la forma con angulos, ese sketch no llegaba siquiera al resolutor de librerias y
moria con 'fatal error: M5Cardputer.h: No such file', mientras el mismo sketch
con angulos compilaba sin problema. Cualquiera que pegue un ejemplo oficial se
daba de bruces con esto.

Se pasan ademas los nombres de los ficheros del propio sketch, para que un
#include "Common.h" local siga siendo una cabecera del proyecto y no se
confunda con una libreria.

test/backend/unit/test_espidf_compiler.py cubre ambas formas, el espaciado, las
cabeceras internas de IDF y la cabecera propia del proyecto.
2026-07-26 02:31:23 +02:00
David Montero Crespo fd6829f645 merge: master (produccion) dentro de v3.2
Las dos ramas habian divergido: master llevaba el modo lenguaje ESP-IDF puro
(#139) y v3.2 la ruta de compilacion IDF v5.5 para toda la familia ESP32 mas
los arreglos de venv/toolchain. Ambas tocaban espidf_compiler.py.

Los dos lados son ejes ORTOGONALES y se conservan enteros:

  - use_idf5 / arduino_mode (v3.2): que arbol IDF usa el build (5.5 vs 4.4) y
    si cabe Arduino-como-componente.
  - pure_idf (master): el modo LENGUAJE que elige el usuario; sus ficheros son
    las fuentes del componente main con su propio app_main().

Resolucion:
  - _build_env acepta los tres. Un build IDF puro fuerza arduino_mode a falso:
    la plantilla CMake mete el componente arduino-esp32 en cuanto existe
    ARDUINO_ESP32_PATH, asi que dejarlo puesto compilaba el core de Arduino en
    un build que no tiene sketch. Lo cazaron los tests de master.
  - VELXIO_PURE_SKETCH solo con pure_idf, nunca con arduino_mode a falso a
    secas: un target sin core arduino-esp32 sigue entregando un SKETCH al
    traductor legacy y no debe tomar la rama del glob puro.
  - La identidad del build-dir suma los dos tokens (|idf:N|ard:N y |lang:pure):
    ningun par de esas combinaciones puede compartir un build/ configurado.
  - La cadena de escritura de fuentes queda pure_idf -> arduino_mode -> legacy.
  - sdkconfig: render de v3.2 (con target/use_idf5) mas el filtrado de simbolos
    CONFIG_ARDUINO* de master cuando el build es puro.

test/backend/unit/test_espidf_compiler.py: los 7 tests que ya estaban rotos en
v3.2 (AttributeError: idf5_path, fixture sin actualizar desde que se anadio la
seleccion de IDF) vuelven a pasar.

Verificado: backend 293 pasan / 0 fallan (v3.2 traia 7 rotos); frontend 2268
pasan / 0 fallan en los dos shards.
2026-07-25 21:52:00 +02:00
David Montero 734b7d0487 feat(esp32): pure ESP-IDF language mode for the ESP32 family (#139)
Adds a third entry to the board language selector next to Arduino C++
and MicroPython: ESP-IDF. In this mode the user writes a plain ESP-IDF
project — app_main() entry point, FreeRTOS + driver APIs — and the
backend compiles it through the same ESP-IDF toolchain it already uses
for ESP32 Arduino sketches, just without the arduino-esp32 component.

Backend:
- CompileRequest.language ('espidf') threaded through the sync + async
  compile paths and folded into the dedup job key (language='arduino'
  and omitted hash identically so old clients keep dedupping).
- espidf_compiler: pure_idf flag. User files are written into main/
  as-is (no Arduino.h wrap, no velxio_compat.h, Arduino library
  resolution skipped), ARDUINO_ESP32_PATH is dropped from the build env
  and VELXIO_PURE_SKETCH raised so the template CMake compiles the
  user's own sources via a glob branch. Pure builds get their own
  persistent build-dir variant through the eff_hash fold.
- QEMU WiFi compat for IDF-style code: esp_wifi.h/esp_wifi_init
  detection sets has_wifi, and literal #define SSID/PASS plus
  wifi_config_t designated initializers are normalized to the QEMU AP.
- CONFIG_ARDUINO_* lines are stripped from sdkconfig.defaults in pure
  mode (the symbols don't exist without the arduino component).

Frontend:
- LanguageMode gains 'espidf'; BOARD_SUPPORTS_ESPIDF covers the ESP32
  family (Xtensa, S3, C3). Toolbar shows the option only for those.
- Switching modes seeds a main.c blink skeleton (app_main + gpio
  driver), mirroring the MicroPython main.py flow.
- compileCode sends language='espidf'; run/stop paths are unchanged
  (the QEMU worker consumes the same merged flash image).
- New gallery example: esp32-idf-blink (LED + resistor on GPIO 2).

Tests: unit coverage for the build-env switch, IDF wifi normalization,
job-key variance, file-group seeding and the new example; verified
end-to-end in a container from the prod image (pure build produces a
bootable flash image; Arduino-mode build unchanged, same variant hash).
2026-07-24 06:37:01 +02:00
David Montero 7db9e41278 test: drop Pico W backend tests (moved to the pro overlay)
test/backend/unit/test_picow_inbound_gateway.py and
test/backend/integration/test_picow_net_bridge.py imported
app.services.picow_net, which moved to the private overlay in the open-core
split. They now live under pro/backend/tests/ and run against the overlay.
Removing them here unbreaks the OSS pytest collection (and the deploy gate).
2026-06-15 08:54:16 +02:00
David Montero 173cc3ea36 feat(picow): IoT gateway — proxy browser HTTP into the chip's server
ESP32 web-server examples are reachable from the browser via
/api/gateway/<client_id>/ (QEMU slirp hostfwd). The Pico W server lives
in the browser-side lwIP, so there was no inbound path: visiting the
chip's IP did nothing.

Add the mirror of tcp_nat.py: tcp_inbound.TcpInbound originates a TCP
connection INTO the chip over the WebSocket bridge (SYN -> SYN+ACK ->
ACK -> request -> response -> FIN), so the backend can fetch a page the
sketch serves on 10.13.37.42:80 and hand it back to the browser.

- bridge.py routes chip TCP segments addressed to a gateway-opened
  connection to TcpInbound (before the chip-initiated NAT, which would
  RST them); exposes http_into_chip() + ensure_chip_mac() (primes the
  chip's gateway ARP).
- iot_gateway.py: same /api/gateway/<client_id>/ route now falls through
  to the Pico W bridge when there's no ESP32 instance, builds a raw
  HTTP/1.1 request, and parses the chip's response. Same plan gate, same
  URL shape — the browser sees no difference between ESP32 and Pico W.

Validated end to end (real RP2040 emulator serving an HTTP page ->
gateway returns it) plus 6 unit tests for the TCP state machine,
response parsing and ARP priming.
2026-06-14 02:15:53 +02:00
David Montero 1d643797b4 fix(esp32): manifest scope resolves to the DECLARED lib, not first-match
P2.0 first cut took the first-alphabetical lib providing a header and then
checked manifest membership. When several installed libs ship the same header
(e.g. DHT118266, DHT_sensor_library, servodht11 all have DHT.h), the stray
first-match got rejected and the header was dropped even though the declared
lib provides it.

_find_manifest_library_for_header: when a manifest is supplied, pick the first
DECLARED library that provides the header. This both selects the right lib and
excludes undeclared ones. No manifest = legacy first-match.

Test strengthened with a stray same-header lib that sorts first.
2026-06-06 09:01:03 +02:00
David Montero 47e220b72f feat(esp32): project library manifest scopes ESP-IDF resolution (P2.0)
When a compile supplies a 'libraries' manifest, _resolve_library_components
merges a USER-installed library only if it's declared in that set. A sketch
therefore never picks up an unrelated library from the shared dir (another
user's install, or a same-named clash) — the manifest is the resolution scope.

- _resolve_library_components(allowed_libraries): gate user-lib merges on
  manifest membership; match by folder name OR library.properties name=,
  normalised (display name vs on-disk folder differ by separators/case).
  Core/bundled libs are never gated. None = legacy scan-all (unchanged).
- Threaded through compile() -> _compile_in_dir.
- compile.py: CompileRequest.libraries; folded into the async dedup _job_key
  so a different manifest doesn't dedup to a job built with another.

Opt-in: omitting 'libraries' preserves current behaviour exactly.
Regression: test/backend/unit/test_espidf_core_first.py::TestManifestScope
2026-06-06 08:46:05 +02:00
David Montero ac3a8a8e4c fix(esp32): core arduino-esp32 headers never resolve to user libs
A user library that ships a core-named header (e.g. WiFiEspAT/src/WiFi.h)
could shadow the arduino-esp32 core during ESP-IDF library resolution.
WiFiEspAT shadowing WiFi.h pulled EspAtDrv.cpp into the build, whose
const char OK[]/STATUS[] collide with ESP-IDF's enum STATUS in
rom/ets_sys.h, breaking every ESP32 sketch that #include <WiFi.h>.

_resolve_library_components now:
- skips a header entirely when the arduino-esp32 core provides it
  (computed set from cores/ + libraries/, cached), so a user lib can
  never shadow WiFi.h/Wire.h/SPI.h/WebServer.h/...
- skips a resolved user lib whose library.properties architectures=
  excludes esp32/*.

Regression: test/backend/unit/test_espidf_core_first.py

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-05 05:11:36 +02:00
davidmonterocrespo24 9f4bf39f65 test(dht22): skip absolute-timing busy_wait checks under CI
test_busy_wait_100us and test_busy_wait_1us measure busy_wait_us()
elapsed time against absolute thresholds (500µs / 100µs). Under
contended CI/deploy-gate machines these can blow through the budget
even when the busy-wait implementation is correct, blocking deploys
that have nothing to do with DHT22 timing.

Same pattern already applied to test_response_timing_analysis in
this file — skipped via @unittest.skipIf(os.environ['CI']=='true').
2026-05-17 22:31:37 +02:00
davidmonterocrespo24 adad446518 fix(esp32): LEDC signal IDs are 71-86 per ESP32 TRM, not 72-87
User report: on the solar-tracker project (5218f9e3) only one servo
moved and the log showed `ch=0 duty=X% gpio=12` (wrong — servoPan was
attached to GPIO 13) and `ch=1 ... gpio=-1` (servoTilt's channel
never resolved).

Root cause traced through the GPIO Matrix dump: the firmware does
exactly what the Arduino-ESP32 Servo library says — `ledcAttachPin(
13, 0)` writes signal 71 (LEDC_HS_SIG_OUT0) into `gpio_out_sel[13]`,
and `ledcAttachPin(12, 1)` writes signal 72 (LEDC_HS_SIG_OUT1) into
`gpio_out_sel[12]`. Per the ESP32 Technical Reference Manual section
4.11, Table 4-3:

    71 .. 78  →  LEDC HS channels 0..7
    79 .. 86  →  LEDC LS channels 0..7

The legacy worker code at esp32_worker.py:426 used the off-by-one
range `72 <= signal <= 87` with `ledc_ch = signal - 72`. The mistake
masked itself for single-servo projects because the 0x5000 duty
callback's channel index was internally consistent with the bogus
math, so the duty STILL reached the correctly-routed pin (just
labelled wrong). The new SignalRouter unit tests caught the
discrepancy the moment two servos drove distinct channels: signal
71 (HS_CH0, gpio 13) was REJECTED by the off-by-one filter and
signal 72 (HS_CH1, gpio 12) was misclassified as channel 0.

When I ported the legacy range into `esp32_signals.SIG_LEDC_HS_CH0_OUT_IDX`
the bug came along for the ride. Fix both modules:

* `backend/app/services/esp32_signals.py`: HS 71-78, LS 79-86.
* `frontend/src/simulation/esp32-signals.ts`: mirror.
* tests updated; 20 backend + 23 frontend pass.

After deploy the user's two servos will resolve to their declared
pins:

    ch=0  duty=X%  gpio=13   (servoPan, was wrongly emitting gpio=12)
    ch=1  duty=X%  gpio=12   (servoTilt, was wrongly emitting gpio=-1)

This is also why the multi-servo blink "patch" in commit 77bf897
appeared to help: with both pins ALIASED to the same channel via
the off-by-one, the broadcast fallback was the only thing producing
ANY movement on the second servo at all.
2026-05-17 05:42:52 +02:00
davidmonterocrespo24 0f05544ca8 feat(esp32): SignalRouter — model the GPIO Matrix as first-class
Replaces the per-peripheral ad-hoc `_ledc_gpio_map` cache with a
proper signal-routing abstraction that mirrors the ESP32 SoC's
IO_MUX + GPIO Matrix exactly. Same idea as real silicon: signal
sources (LEDC channels, RMT, MCPWM, ...) → 40-entry routing table
→ GPIO pins.

Motivation (from user bug report in
velxio.dev/project/5218f9e3-136d-43b3-bba1-6cebde21e1a4): two
ESP32 servos on a solar-tracker visibly oscillated between two
positions instead of moving smoothly when the user changed LDR
sliders. Commit 77bf897 patched it (per-channel gpio memo +
broadcast guard) but the user requested a proper hardware-fidel
architecture, not patches.

Backend:
* `app/services/signal_router.py` — SignalRouter class. Forward
  index (gpio → signal_id) + reverse index (signal_id → set of
  gpios). `replace_snapshot()` returns the diff for the polling-
  fallback path; future C plugin hook becomes a push without
  touching this code.
* `app/services/esp32_signals.py` — Signal id constants from
  ESP32 TRM (LEDC HS 72-79, LS 80-87) + `ledc_signal_for_channel()`
  helper.
* `app/services/esp32_worker.py` — `_ledc_gpio_map` is gone;
  `_refresh_ledc_gpio_map` replaced by `_refresh_signal_routing`
  which emits `gpio_routing {gpio, signal_id}` events on diff.
  The 0x5000 LEDC callback and the LEDC poll thread now emit
  `ledc_duty {channel, duty_pct}` (canonical, no gpio) alongside
  the legacy `ledc_update {channel, duty, gpio}` for back-compat
  during rollout.

Frontend:
* `simulation/SignalRouter.ts` — 1-to-1 TS mirror of the Python
  class. Same forward + reverse index; same `pinsForSignal` /
  `updateRouting` / `clearRouting` API.
* `simulation/esp32-signals.ts` — Signal id constants, mirror
  of the Python module.
* `simulation/Esp32Bridge.ts` — new `onLedcDuty`, `onGpioRouting`,
  `onGpioRoutingClear` callbacks; handlers for the new event types.
* `store/useSimulatorStore.ts` — `makeLedcDutyHandler` looks up
  pins via `router.pinsForSignal(ledcSignalForChannel(channel))`
  and dispatches per pin. `makeGpioRoutingHandler` /
  `makeGpioRoutingClearHandler` keep the mirror in sync. Per-board
  `signalRouterMap` parallels `pinManagerMap` in lifecycle.
  `makeLedcUpdateHandler` (and its memo workaround from 77bf897)
  stays wired for back-compat during rollout; removed in a
  follow-up commit once prod is verified stable on the new path.

Tests:
* `test/backend/unit/test_signal_router.py` (20 tests) covers
  update/clear semantics, idempotency, multi-pin routing,
  snapshot diff, channel↔signal-id helpers, and the multi-servo
  regression scenario.
* `frontend/src/__tests__/SignalRouter.test.ts` (17 tests) is the
  mirror — same scenarios on the TS side.
* `frontend/src/__tests__/esp32-multi-servo-gpio-matrix.test.ts`
  (6 tests) drives the end-to-end SignalRouter handler pipeline,
  asserts that two servos on GPIO 13/12 via LEDC channels 0/1
  move independently (no mirroring), that re-routing carries
  cleanly, and — critically — that `PinManager.broadcastPwm` is
  never called.

Totals: +700 LOC, 1876 frontend tests pass (was 1853), 278 backend
unit tests pass (was 259).

Docs: ESP32_EMULATION.md §9.2 rewritten with the new architecture
diagram + a runbook for adding future peripherals through the
SignalRouter.

The C plugin hook in qemu-lcgamboa that would push gpio_out_sel
writes synchronously (eliminating the polling race window entirely)
is the next step — kept as a follow-up because the polling-fallback
path here already resolves the routing before each duty event
fires, so the bug is fixed end-to-end. The plugin work removes the
race condition fundamentally.
2026-05-17 05:00:53 +02:00
davidmonterocrespo24 9d26fa6de3 test(dht22): skip the microsecond-timing analysis under CI=true
test_response_timing_analysis measures the actual µs duration of the
DHT22 preamble LOW pulse and asserts it stays under 1000µs (real
hardware target ~80µs, busy-wait tolerance ~500µs). On a deploy box
under load (concurrent docker build + zstd compression + container
runtime) GIL contention inflates the observed timing far past the
threshold — the deploy gate just hit 3242µs and aborted.

Mirror the same @skipIf(CI=='true') gate the sibling
test_response_data_matches_payload already has (line 390-393).
Locally / when debugging the DHT22 path the test still runs in full.
2026-05-17 03:56:43 +02:00
davidmonterocrespo24 04ac1bf53b chore(tests): silence three noisy warnings in deploy-gate output
deploy.sh's vitest + pytest output was polluted with three benign
but loud warnings that buried real signal:

1. AVRSimulator.start() unconditionally read `window.__spiceDebug`.
   In node-side vitest runs `window` is undefined → ReferenceError
   → console.warn('[spice] debug dump failed', e). Logged once per
   AVR test. Guarded with `typeof window !== 'undefined'`; in
   production the browser path is unchanged.

2. pinPositionCalculator.calculatePinPosition() warned every time
   document.getElementById returned null. In node-side tests there
   is no real DOM and every wire-related test triggers the warning
   for every component. Skip the console.warn when
   import.meta.env.MODE === 'test' (vitest sets MODE=test); the
   function still returns null and production retains the
   actionable warning for unmounted components.

3. test_esp32_wifi_args.py::test_start_instance_accepts_wifi_params
   mocked asyncio.create_task with no side_effect, so the coroutine
   from self._boot(...) leaked and triggered a "coroutine never
   awaited" RuntimeWarning. Mock now closes the coroutine.

After fixes:
  frontend tests:  0 spice/pinPositionCalculator stderr lines
  backend tests:   259 passed, 15 skipped, 1 warning (starlette
                   third-party python_multipart deprecation —
                   not ours, fixed when starlette updates).
2026-05-16 22:39:43 +02:00
davidmonterocrespo24 b9d39c0bd7 fix(pi3): show kernel boot + autologin SD + sidecar cache invalidation
User report: clicked Pi 3 board → nothing visible happens. Three
defects, all on the same path:

1. The kernel cmdline carried over from the original pre-OSS-split
   code: `quiet init=/bin/sh`. Result: kernel boot messages
   suppressed, then dropped straight to bare /bin/sh with no PS1 so
   the user sees an empty serial. Removed both. The kernel cmdline
   is now just `console=ttyAMA0 root=/dev/mmcblk0p2 rootwait rw
   dwc_otg.lpm_enable=0`, which lets systemd start a real
   serial-getty@ttyAMA0.service.

2. Pi OS Trixie armhf since Bookworm ships without a default user
   (no more pi/raspberry). With cmdline #1 fixed, the user would
   land at a login prompt and be stuck. Fix: pre-bake a systemd
   drop-in at /etc/systemd/system/serial-getty@ttyAMA0.service.d/
   autologin.conf that uses `agetty --autologin root` so the serial
   console drops to a root shell on first prompt. The browser
   canvas IS the authentication boundary; the SD image is mounted
   RO via a qcow2 overlay so per-session edits don't persist.
   Edit happens in velxio-prod/scripts/configure-pi3-autologin.sh
   (to follow in a separate commit).

3. Architectural: the original cache-hit probe was size-only.
   Today's SD image rebake produced a file with identical byte count
   but different SHA256 — the cache served stale content for every
   request even after a manifest bump. Fix: write a sidecar
   `<file>.sha256` after every successful materialise and trust it
   on subsequent probes. Manifest SHA bumps invalidate the cache
   regardless of size. Two regression tests guard this:
     - test_provider_sidecar_invalidates_on_sha_mismatch
     - test_provider_missing_sidecar_treats_file_as_invalid

Manifest bumped to version "2026-04-21+autologin" for the SD image
(kernel + DTB unchanged, still 2026-04-21).
2026-05-16 06:23:22 +02:00
davidmonterocrespo24 93fd4617af feat(sim): boot_images module + Pi 3 emulation restored
Pi 3 simulation had been broken since at least April 2026 (51
fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct
defects compounded:

1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree
   blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS
   SD image — none of which shipped in the repo or were pulled at
   image build.

2. qemu-system-arm + qemu-utils were missing from the Docker image
   entirely, so even with the boot files in place QEMU couldn't
   launch. Add both to Dockerfile.standalone (~200 MB).

The architecture fix is a new `app.services.boot_images` module:

  * Manifest-driven (boot_images/manifest.json, versioned in repo,
    declares SHA256 + size for each file, supports an optional
    `compressed.{encoding,sha256,size_bytes}` block for assets shipped
    as .zst).

  * `BootImageProvider` materialises files lazily, atomically (temp +
    rename), verifies SHA256 pre- AND post-decompression, caches under
    /var/cache/velxio/boot-images, serialises concurrent get() calls
    per image set via asyncio.Lock.

  * `AssetDownloader` Protocol with two impls:
    - `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs
      use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY).
    - `LocalDirectoryDownloader` — for tests + in-prod use where the
      licence-module storage is already on the same filesystem (saves
      the loopback HTTP roundtrip on a 1.4 GiB blob).

  * `build_downloader_from_env()` picks one — local-dir wins if both
    sets of env vars are present, so the prod box short-circuits to
    direct disk reads automatically.

  * Lifespan hook in qemu_manager.py pre-warms the cache on container
    boot so first-time user requests don't pay the 30-60 s download
    + decompress latency.

Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via
upload-binary.sh, append an entry to manifest.json, register a
lifespan pre-warm in the new board's service module. Zero edits to
provider.py / downloader.py.

Manifest entries for raspberry-pi-3:
  kernel8.img             9 695 883 bytes  (uncompressed)
  bcm2710-rpi-3-b.dtb        34 687 bytes  (uncompressed)
  raspios-trixie-armhf.img  5 729 419 264 bytes raw
                          / 1 488 002 803 bytes .zst on wire (zstd -19)
  source: 2026-04-21 build from raspberrypi.com

Tests: 21 new unit tests covering manifest parsing, integrity
helpers, both downloaders, and the provider's idempotent /
concurrent / integrity / decompression / warmup paths. In-process
FakeDownloader keeps the suite under 1 s and httpx-free.

Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk
layout, named-volume operation, and the procedure for adding a new
image set.
2026-05-16 05:41:46 +02:00
davidmonterocrespo24 c2fe1af250 perf(compile): dedup, concurrency limits, and persistent build dir for ESP-IDF
Three coordinated fixes that together close the "ESP-IDF compile takes
5-7 min every time" gap and prevent the failure mode where a user clicking
compile multiple times spawns six ninja processes that peel each other
apart on a modest VPS.

What was wrong
- /compile/start generated a fresh uuid4 every call, so 6 clicks = 6
  independent builds racing each other. Saw load average 30 on the prod
  VPS during a real BMP280 attempt today.
- No concurrency limit anywhere; asyncio.create_task() fired without
  gating.
- ccache was wired in last week (PR #149) but reported 18,350 cacheable
  calls and **0 hits** because the build dir was a fresh
  tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random
  /tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags
  → different command line every compile → ccache hash miss every time.

What this PR does

1. Job deduplication (`backend/app/api/routes/compile.py`)
   - New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file
     names + contents + board. Order-independent.
   - New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id.
   - `compile_start` checks JOB_BY_KEY before spawning a new task; if a
     job for this exact content is already pending or running, returns
     the existing job_id (logs `[compile] dedup hit — reusing job <id>`).
   - `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY,
     keeping the index consistent. Edge case where two jobs share a key
     (old finished, new running) is handled — only evict the key entry
     if it still points at the purged job.

2. Concurrency control (`backend/app/api/routes/compile.py`)
   - `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on
     simultaneous compiles.
   - `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so
     concurrent compiles to the SAME board (sharing the persistent build
     dir) serialise. Different boards still run in parallel up to the
     semaphore cap.
   - `_compile_job` acquires sema → per-target lock → flips state to
     `running` → calls `_run_compile`. Pending state now accurately
     reflects "queued waiting for resources".

3. Persistent build dir (`backend/app/services/espidf_compiler.py`)
   - New `_prepare_persistent_project_dir(idf_target)` materialises
     `/var/lib/velxio-build/<target>/project/` from the template on
     first use; on subsequent compiles it wipes only `main/` and
     `user_libs/` (the per-compile parts) and leaves `build/` alone so
     ninja's incremental cache + ccache .o files survive.
   - Toolchain version sentinel (`.idf_version`) wipes the whole target
     dir if the ESP-IDF or arduino-esp32 version changes — cached
     objects from the old toolchain are no longer ABI-compatible.
   - `compile()` is now a thin dispatcher: persistent path or fallback
     to the legacy `tempfile.TemporaryDirectory()` flow. The actual
     build logic was extracted into `_compile_in_dir()` so both paths
     share one implementation, no duplication.
   - Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back
     to the tempfile path without rebuilding the image. Critical for
     production safety.

4. ccache normalisation (`Dockerfile.standalone`)
   - + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise
     absolute paths under that prefix when computing the cache key.
     Robustens hits against any future subdir rearrangement.

5. Docker compose (`docker-compose.yml`)
   - + named volume `velxio-build:/var/lib/velxio-build` so the persistent
     build dir survives `docker compose up -d --build`.
   - + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable
     without rebuilding).

Expected impact
- Cold first compile per container per target: unchanged (~5-7 min).
- Same sketch re-compiled: ~2-5 s (everything cached).
- Different sketch, same target: ~5-30 s (only user code + new lib steps
  rebuild; ESP-IDF base hits cache).
- Different sketch with new libraries: ~30-90 s (new lib component
  compiles; rest hits cache).
- Concurrent clicks on same example: 1 build, others poll the same
  job_id. No more six-ninja meltdown.

Tests
- `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability +
  variance and `_purge_expired_jobs` consistency (including the
  "two jobs share a key" edge case).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 08:33:11 +02:00
David Montero Crespo 26c7d50310 fix(ci): two stale-path bugs from the recent refactors
1. Backend test (test_arduino_cli_attinycore.py): the entrypoint script
   was renamed deploy/ → docker/ in commit b736aea but this test still
   pointed at the old path. Update the read_text() call + docstring.

2. Frontend CI (frontend-tests.yml): the cache key
   `frontend-${{ hashFiles('frontend/package-lock.json') }}` was tied to
   a file that has since been gitignored (commit eb9a3ec). hashFiles()
   on a missing file returns the same empty hash forever, so every CI
   run was restoring the same stale node_modules — including the
   symlinks to `file:../third-party/wokwi-elements` that existed before
   the npm migration in commit 531c337. On revalidation, npm tried to
   run wokwi-elements' `prepare` script (`husky install && npm run
   build`), which failed with "husky: not found".

   Drop the cache step entirely; lock files aren't committed so cache
   keys can't be made meaningful without overcomplication. Adds ~30s
   per CI run, but actually correct. Also pass --no-audit --no-fund
   to npm install for cleaner logs.
2026-05-05 11:22:02 -03:00
David Montero Crespo 641ac8c1de Add comprehensive tests for Cyw43Emulator functionality and lifecycle
- Implemented handshake tests to validate initial bus state and register responses.
- Created end-to-end tests for Pico W LED blinking using MicroPython firmware.
- Added SDPCM framing tests to ensure proper encoding and decoding of control frames.
- Developed IOCTL tests to verify command responses and state changes in the emulator.
- Established a full lifecycle test for WiFi operations, including scanning, connecting, and packet handling.
- Introduced TypeScript configuration for test files to ensure compatibility and strict type checking.
2026-04-29 00:21:26 -03:00
David Montero Crespo 07c9718bc3 test: update BMP280 chip ID assertions and clarify tagName uniqueness in metadata drift test 2026-04-21 17:31:51 -03:00
David Montero Crespo d4a48e6e1e feat: Skip real paths test in CI; mark subproject commits as dirty 2026-04-11 15:49:32 -03:00
David Montero Crespo 382e13cfe7 feat: Update components metadata timestamp and mark subproject commits as dirty; add diagnostic test for real library paths 2026-04-11 15:42:38 -03:00
David Montero Crespo 2ba8020438 feat: Enhance ESPIDFCompiler library resolution logic; add support for dynamic library detection and patching in CMakeLists.txt
refactor: Update wiring examples for E32 OLED integration; correct pin mappings for VCC, GND, DATA, and CLK
test: Improve unit tests for ESPIDFCompiler; add scenarios for library resolution and CMake patching
chore: Mark subproject commits as dirty for wokwi-libs
2026-04-11 15:25:40 -03:00
David Montero Crespo 6b161eab34 feat: Enhance CI workflows for backend unit and e2e tests; add environment setup and skip timing-sensitive tests in CI 2026-04-10 23:47:25 -03:00
David Montero Crespo 7971743174 Add unit tests for I2C slave devices, MCP tools, and WiFi/BLE status parser
- Implement tests for BMP280, DS1307, DS3231, I2CWriteSink, and MPU6050 slaves in test_i2c_slaves.py.
- Create test suite for Velxio MCP server tools in test_mcp_tools.py, covering Wokwi utilities and circuit management functions.
- Add tests for parsing WiFi and BLE serial output in test_wifi_status_parser.py, ensuring correct status events are captured.
2026-04-10 23:39:51 -03:00