End-to-end pipeline fixes uncovered while auditing the /examples gallery.
Each bug shipped past green unit + snapshot tests because none of those run
firmware + render LEDs. Added scripts/visual-led-test.mjs as a CDP-driven
visual harness that loads each example, runs the simulator, samples
`wokwi-led.brightness`, and asserts toggle / gradient / initial-off
invariants — exits non-zero on any regression.
Frontend simulator
- PinManager.updatePort: new optional ddrMask param. A pin is added to
`outputPins` only if the DDR bit is set, so the PORTx write that
enables INPUT_PULLUP (DDR=0, PORT=1) no longer falsely marks the pin
as MCU output. AVRSimulator now reads DDRB/C/D (0x24/0x27/0x2A on
Uno/Nano, 0x37 on ATtiny85, per-port table on Mega) and forwards it.
- AVRSimulator: pass DDR mask alongside every port-listener fire.
- BasicParts pushbutton{,-6mm}: seed pin HIGH in attachEvents so
`digitalRead()` returns HIGH while idle. avr8js doesn't auto-simulate
INPUT_PULLUP — without this the firmware reads LOW from boot and
thinks the button is permanently pressed (the "LED is always on,
pressing does nothing" UX bug).
- connectMcuEdgesToService: suppress synthetic digital edges on pins
with active PWM, AND subscribe to onPwmChange to re-tick the netlist
on duty changes. Fade-LED now produces a true gradient (6 distinct
brightness levels across a fade cycle) instead of a binary 0/full
toggle.
- CircuitSimulationService.handleMcuEdge: replace single-slot
pendingMcuEdge with a per-pin Map. Multiple pins toggling during the
same in-flight tick used to overwrite each other; now every pin's
most-recent edge replays after the tick. Fixes Traffic-Light RED→
YELLOW→GREEN sequencing.
- NetlistBuilder: new sanitizeSpiceId() helper replaces hyphens with
underscores in V-source names. ngspice's interactive `alter` command
treats `-` as an operator and silently no-ops on hyphenated source
names, so mid-simulation MCU pin transitions stopped propagating
after the first solve. MixedModeScheduler.onMcuPinChange and
CircuitSimulationService self-heal use the same sanitizer so names
stay consistent across emit/alter/lookup. Also added a regex-based
fallback in step 2 so any board pin matching `GND.\d+` canonicalises
to net "0" — ESP32-C3 dev kits expose up to 10 GND pins and the
per-board `groundPinNames` list missed several, leaving wires
floating instead of grounded.
- collectPinStates: emit V-sources only for pins in `outputPins`, not
every wired board pin. Leaves INPUT pins (analog sensors on A0,
pull-down dividers, etc.) free for the SPICE solver instead of being
shorted to 0 V by an ideal MCU V-source.
- start.ts: extended __spiceDebug to also expose outputPinsByBoard +
nodeVoltages + pinNetMapEntries for the visual harness.
- ESP32 / RP2040 / RISC-V / C3 simulators: pass `'mcu'` source flag to
triggerPinChange / setPinState so the new outputPins tracking fires
on those boards too (was AVR-only before).
- useSimulatorStore: stopBoard/resetBoard call pm.resetPinStates() so
outputPins clears between runs; Esp32Bridge.onPinChange passes the
`'mcu'` flag in all three places it's wired.
- types/board.ts: ATtiny85 FQBN `clock=internal16mhz` →
`clock=16pll` (ATTinyCore 1.5.2 renamed the option).
Backend
- esp-idf-template/main/CMakeLists.txt: skip the
`-DLED_BUILTIN=2` fallback for esp32c3 and esp32s3 targets. Both
variants already define LED_BUILTIN in pins_arduino.h via a
self-define macro (`#define LED_BUILTIN LED_BUILTIN` + `static const
uint8_t LED_BUILTIN = ...;`). Pre-defining the symbol from the
command line expanded the static-const declaration to
`static const uint8_t 2 = ...;` — a syntax error that broke every
ESP32-C3 / S3 build (`expected unqualified-id before numeric
constant`).
Examples
- examples.ts: bulk-fix 72 wire endpoints that referenced
`componentId: 'nano-rp2040'` / `'esp32-c3'` etc. (boards that don't
exist on the canvas). Replaced with `'arduino-uno'` (the canvas
board-id convention) and converted `D<n>` pin names to `GP<n>` for
Pico-style boards. Affects pico-blink, pico-i2c-scanner,
pico-i2c-rtc-read, pico-spi-loopback, c3-blink and others.
Tests
- scripts/visual-led-test.mjs: CDP-driven harness. Default suite covers
Blink (single-pin), Button (idle-OFF invariant — catches the
INPUT_PULLUP regression), Traffic-Light (multi-pin sequencing),
Fade-LED (PWM gradient — ≥3 distinct levels), RGB-LED (≥3 PWM pins
driven). Run via `npm --prefix frontend run test:visual` against a
Chrome on `:9222` + vite on `:5174` + backend on `:8001`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Backend:
- api/routes/compile.py accepts board-specific compile options
and dedups in-flight identical requests
- services/espidf_compiler.py expanded ESP-IDF wrapper with the new
options surface (sdkconfig.defaults.in
template added)
- services/arduino_cli.py honour the new options envelope
- services/esp32_lib_bridge.py thread board options through to QEMU
Tests:
- tests/test_compile_request_dedup.py end-to-end dedup behaviour
- tests/test_espidf_options.py covers the new options parsing
Frontend:
- services/compilation.ts client-side mirror — sends the new
options field on every compile request
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the deferred Phase 3.3. Root-causes the Pi 2 "Attempted to
kill init" panic as `mount /dev/vda` failing with EINVAL — Debian
armmp does not have ext4 builtin (only fuseblk in /proc/filesystems).
- qemu_manager: PI_CONFIGS gains raspberry-pi-zero / -1 / -2 entries.
All three use the armmp armhf kernel + Cortex-A7 CPU + the mmio
virtio transport (arm-32 virt PCI fails -75 due to missing reg DT
property). Pi Zero / Pi 1 get the small 1-core / 512 MB profile;
Pi 2 gets 4-core / 1 GB. QEMU command builder branches on cfg.bus
for virtio-blk-pci vs virtio-blk-device (and serial likewise).
- manifest.json: new `raspberry-pi-armhf` image_set wiring three
assets (kernel + initramfs + zstd rootfs).
- Frontend BoardKind gains the three new kinds + an isPiBoardKind()
helper. Replaces the eight scattered `=== 'raspberry-pi-3' ||
=== 'raspberry-pi-4' || === 'raspberry-pi-5'` branches in
useSimulatorStore, Interconnect, loadExample, boardProtocols.
ComponentRegistry gets three new picker entries.
- board-kinds-coverage test: ACCEPTED_UNCOVERED gains the new kinds
(backend boards have no canvas examples).
The matching armhf build-pi-kernel.sh / build-pi-rootfs.sh changes
live in velxio-prod's scripts/ (private overlay) — the upstream
kernel build script only knows about arm64; armhf is built in the
private repo because the assets ship through the license endpoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the public extension points the velxio-prod overlay uses to bind
real canvas-side I2C/SPI/UART models (BME280, future MCP23017, etc.)
to a running Pi guest's protocol shims:
- qemu_manager: set_pi_slave_handler(fn) / get_pi_slave_handler() for
pi_attach_slave + pi_detach_slave WebSocket messages. OSS image
leaves the hook unset so the messages are silently dropped.
- simulation route: parses the two new WS message types and forwards
them to the registered handler when present.
- RaspberryPi3Bridge: attachSlave(spec) / detachSlave(spec) frontend
side of the protocol.
- piSlaveScanner: at simulation start walks components + wires,
identifies I2C/SPI/UART peers wired to Pi protocol pins (40-pin
header physical-pin numbering), and emits one attach per
bus/address pair (deduped across SDA+SCL wires).
- RaspberryPiWorkspace: invokes the scanner once the bridge is open,
with retries to ride out the WS-still-connecting race.
- integration test: pi3_bme280_attach.py boots the Pi, pre-attaches a
BME280 via the slave handler, runs a host-side proto loop, runs
guest python smbus2.read_byte_data(0x76, 0xD0) and asserts the
console reads back CHIP=0x60.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend: extract per-board config into a PI_CONFIGS dict keyed by
board_type. Pi 3/4/5 share the same arm64 image set (kernel +
initramfs + rootfs) and differ only in QEMU -cpu and -m:
raspberry-pi-3 → cortex-a53 + 1G (BCM2837, ARMv8 64-bit)
raspberry-pi-4 → cortex-a72 + 2G (BCM2711, ARMv8 64-bit)
raspberry-pi-5 → cortex-a76 + 2G (BCM2712, ARMv8 64-bit)
PiInstance now carries board_type so the per-board lookup happens
once at start_instance time. Unknown board_type falls back to
DEFAULT_PI_BOARD ('raspberry-pi-3') instead of erroring out (for
back-compat with older clients).
Pre-warm hook walks every unique image_set in PI_CONFIGS so the
provider only downloads each set once even when several Pi models
are registered.
Frontend:
- BoardKind union gains 'raspberry-pi-4' and 'raspberry-pi-5'.
- BOARD_KIND_LABELS + BOARD_KIND_FQBN entries for both new boards
(FQBN null since they use the Pi VFS + Python toolchain like Pi 3).
- ComponentRegistry inserts two new component metadata entries
cloning the Pi 3 board art with different thumbnail colours.
Tag name reused so the same velxio-raspberry-pi-3 web element
draws the board on the canvas — the 40-pin GPIO layout is
identical across Pi 3/4/5.
- boardProtocols.ts: Pi 3/4/5 share the BCM physical→GPIO table
(PI3_BCM) since the 40-pin header layout is identical.
- loadExample.ts: where 'raspberry-pi-3' is special-cased (VFS
ingest, .cpp vs .ino filename), now matches Pi 3/4/5 alike.
- Interconnect.isPi3Bridge() recognises all three Pi family members
so Arduino↔Pi serial routing keeps working.
- RaspberryPi3Bridge constructor gained a boardKind parameter
defaulting to 'raspberry-pi-3'. The WebSocket 'start_pi' message
now ships the actual board kind so the backend knows which
PI_CONFIGS entry to use.
- useSimulatorStore.addBoard wires bridge construction for all
three Pi family members.
Pi Zero/Pi 1/Pi 2 (armhf) come in Phase 3.3 — separate kernel
package + armhf rootfs build, no change here.
Smoke-tested inside the prod container:
Pi 4 (cortex-a72) → reached agetty login on hvc0
Pi 5 (cortex-a76) → reached agetty login on hvc0
Both show 'aarch64' in uname -m.
The Phase 2 E2E test was sending the Python GPIO command via
'python3 -c "..."' but bash quote-nesting silently corrupted the
script — the python process started, printed nothing, exited 0, and
the test asserted 'GPIO_SETUP 17 out' was missing in proto bytes
(it never got sent because the python script never ran).
Switch the test to base64-encode the script + pipe through base64 -d
into a file, then execute. Verified end-to-end now:
[test] proto received 36 bytes:
GPIO_SETUP 17 out pud_off
GPIO 17 1
[test] ✓ shim → proto pipeline works
Also bump the rootfs manifest entry to the final Phase 2 build
(d6d4a274 raw / debd1c33 zst, version 2026.05+phase2-shims-final).
Earlier auto-discovery in _transport.py was hanging at import time
on some glob/sysfs interaction. Now hardcoded /dev/vport1p1 which
is the empirical path under -M virt + virtio-blk-pci on slot 0.
QEMU 10's virtserialport on a socket chardev (server=on,wait=off)
silently drops guest→host bytes. Reproduced cleanly: writes from
inside the guest to /dev/vport<N>p<M> succeed (no errno) but the
connected client socket receives 0 bytes. Same bug whether the
client is a single recv loop, multiple threads, TCP or UNIX socket,
or whether QEMU runs as server vs client. virtconsole on the same
socket works fine — only virtserialport is broken.
Workaround: use `pipe` chardev (a pair of named FIFOs created
beforehand by qemu_manager). guest→host through .out flows reliably
in QEMU 10 — verified with manual test: 'echo PIPE_TEST > /dev/vport1p1'
in the guest produces 'PIPE_TEST\n' immediately on the host side.
Changes:
- qemu_manager._boot: allocate a temp basename, mkfifo .in + .out,
pass to QEMU as 'pipe,path=<base>'.
- qemu_manager._connect_gpio: open both FIFOs O_RDWR | O_NONBLOCK on
host side (O_RDWR keeps the FIFOs open even when guest hasn't
opened its side yet), wire .out into asyncio via loop.add_reader.
- qemu_manager._reply_gpio / _send_gpio: write to .in fd via os.write.
- qemu_manager._handle_gpio_line: extended Phase 1 GPIO-only parser
into a full Phase 2 mux: GPIO/GPIO_SETUP/GPIO_IN/PWM_*/I2C/SPI/UART
with appropriate replies.
- qemu_manager._shutdown: close FDs + unlink the FIFOs.
- manifest.json: bump raspberry-pi-3-virt rootfs to 2026.05+phase2-shims
(the new rootfs ships the velxio shim Python modules under
/usr/lib/velxio-shims/).
raspi3b pl011 RX is broken in QEMU 10 + kernel 6.12 — see
project/pi-emulation/decisions.md for the full debugging trail.
This commit lands Phase 1 of the rebuild: switch the QEMU machine
to virt + cortex-a53, boot the velxio kernel/initramfs/rootfs over
virtio-blk-pci, and expose the user shell on /dev/hvc0 via
virtio-serial-pci + virtconsole.
End-to-end smoke verified: boot → agetty autologin → bash prompt →
echo round-trip returns the typed token. Tested inside the prod
container with QEMU 10.0.8 and our cloud-derived kernel 6.12.88.
What changed:
backend/app/services/qemu_manager.py
PI3_IMAGE_SET -> raspberry-pi-3-virt
PI3_KERNEL_NAME / PI3_INITRAMFS_NAME / PI3_ROOTFS_NAME new
QEMU cmd rewritten end-to-end:
-M virt -cpu cortex-a53 -smp 4 -m 1G
-kernel <velxio-kernel-arm64> -initrd <velxio-initramfs-arm64.cpio.gz>
-drive ... -device virtio-blk-pci (NOT virtio-blk-device — mmio
variant left /dev/vda unregistered)
-nic none -display none -monitor none -serial none
-chardev socket... -device virtio-serial-pci -device virtconsole
(user console -> /dev/hvc0)
-chardev socket... -device virtserialport,name=velxio-protocol
(Phase 2 channel -> /dev/vport0p2)
No -dtb (virt generates its own), no -append init=... (kernel runs
our initramfs which then switch_root to rootfs and exec's its
/sbin/init — Alpine OpenRC).
backend/app/services/boot_images/manifest.json
New image set raspberry-pi-3-virt with three assets uploaded via
the existing license-endpoint pipeline. Old raspberry-pi-3 entry
flagged deprecated:true and kept for one release for rollback.
test/pi3_console_boot/test_pi3_console_boot.py
Updated QEMU argv to match qemu_manager exactly. Markers now look
for the Velxio Pi Simulator MOTD + 'login on hvc0' (autologin
proof). Round-trip echo still required to pass.
PiTerminal didn't call term.focus() on mount, so xterm.js stayed
passive — onData only fires when the DOM element has focus. Users
saw the boot prompt but their keystrokes went to whatever element
held focus when they clicked Run (canvas, code editor), never
reaching the bridge. Calling focus() right after fit() makes the
prompt receive input the moment it's visible.
The qemu_manager change adds INFO-level logging when serial_input
WebSocket messages reach send_serial_bytes — useful diagnostic for
future Pi3 input problems (proves whether bytes reached the backend
before we look at TTY / kernel / PL011 wiring).
User report: on the solar-tracker project (5218f9e3) only one servo
moved and the log showed `ch=0 duty=X% gpio=12` (wrong — servoPan was
attached to GPIO 13) and `ch=1 ... gpio=-1` (servoTilt's channel
never resolved).
Root cause traced through the GPIO Matrix dump: the firmware does
exactly what the Arduino-ESP32 Servo library says — `ledcAttachPin(
13, 0)` writes signal 71 (LEDC_HS_SIG_OUT0) into `gpio_out_sel[13]`,
and `ledcAttachPin(12, 1)` writes signal 72 (LEDC_HS_SIG_OUT1) into
`gpio_out_sel[12]`. Per the ESP32 Technical Reference Manual section
4.11, Table 4-3:
71 .. 78 → LEDC HS channels 0..7
79 .. 86 → LEDC LS channels 0..7
The legacy worker code at esp32_worker.py:426 used the off-by-one
range `72 <= signal <= 87` with `ledc_ch = signal - 72`. The mistake
masked itself for single-servo projects because the 0x5000 duty
callback's channel index was internally consistent with the bogus
math, so the duty STILL reached the correctly-routed pin (just
labelled wrong). The new SignalRouter unit tests caught the
discrepancy the moment two servos drove distinct channels: signal
71 (HS_CH0, gpio 13) was REJECTED by the off-by-one filter and
signal 72 (HS_CH1, gpio 12) was misclassified as channel 0.
When I ported the legacy range into `esp32_signals.SIG_LEDC_HS_CH0_OUT_IDX`
the bug came along for the ride. Fix both modules:
* `backend/app/services/esp32_signals.py`: HS 71-78, LS 79-86.
* `frontend/src/simulation/esp32-signals.ts`: mirror.
* tests updated; 20 backend + 23 frontend pass.
After deploy the user's two servos will resolve to their declared
pins:
ch=0 duty=X% gpio=13 (servoPan, was wrongly emitting gpio=12)
ch=1 duty=X% gpio=12 (servoTilt, was wrongly emitting gpio=-1)
This is also why the multi-servo blink "patch" in commit 77bf897
appeared to help: with both pins ALIASED to the same channel via
the off-by-one, the broadcast fallback was the only thing producing
ANY movement on the second servo at all.
The esp32_worker.py subprocess is launched via `python <abs_path>`
and runs with a sys.path that does NOT include the backend/ package
root, so `from app.services.signal_router import SignalRouter`
raised ModuleNotFoundError at worker startup. The worker exited
with code 1 before QEMU even loaded, and the frontend surfaced the
generic "ESP32 crash detected — cache error" banner.
Mirror the existing esp32_flash_image fallback pattern (already in
this same file): try the package import first, fall back to
importlib.spec_from_file_location with the sibling .py path, then
publish the resulting module under its bare name in sys.modules so
typing references continue to work.
Verified: a synthetic test that strips backend/ from sys.path can
still construct a SignalRouter via the fallback. 20 unit tests in
test_signal_router.py still pass.
Replaces the per-peripheral ad-hoc `_ledc_gpio_map` cache with a
proper signal-routing abstraction that mirrors the ESP32 SoC's
IO_MUX + GPIO Matrix exactly. Same idea as real silicon: signal
sources (LEDC channels, RMT, MCPWM, ...) → 40-entry routing table
→ GPIO pins.
Motivation (from user bug report in
velxio.dev/project/5218f9e3-136d-43b3-bba1-6cebde21e1a4): two
ESP32 servos on a solar-tracker visibly oscillated between two
positions instead of moving smoothly when the user changed LDR
sliders. Commit 77bf897 patched it (per-channel gpio memo +
broadcast guard) but the user requested a proper hardware-fidel
architecture, not patches.
Backend:
* `app/services/signal_router.py` — SignalRouter class. Forward
index (gpio → signal_id) + reverse index (signal_id → set of
gpios). `replace_snapshot()` returns the diff for the polling-
fallback path; future C plugin hook becomes a push without
touching this code.
* `app/services/esp32_signals.py` — Signal id constants from
ESP32 TRM (LEDC HS 72-79, LS 80-87) + `ledc_signal_for_channel()`
helper.
* `app/services/esp32_worker.py` — `_ledc_gpio_map` is gone;
`_refresh_ledc_gpio_map` replaced by `_refresh_signal_routing`
which emits `gpio_routing {gpio, signal_id}` events on diff.
The 0x5000 LEDC callback and the LEDC poll thread now emit
`ledc_duty {channel, duty_pct}` (canonical, no gpio) alongside
the legacy `ledc_update {channel, duty, gpio}` for back-compat
during rollout.
Frontend:
* `simulation/SignalRouter.ts` — 1-to-1 TS mirror of the Python
class. Same forward + reverse index; same `pinsForSignal` /
`updateRouting` / `clearRouting` API.
* `simulation/esp32-signals.ts` — Signal id constants, mirror
of the Python module.
* `simulation/Esp32Bridge.ts` — new `onLedcDuty`, `onGpioRouting`,
`onGpioRoutingClear` callbacks; handlers for the new event types.
* `store/useSimulatorStore.ts` — `makeLedcDutyHandler` looks up
pins via `router.pinsForSignal(ledcSignalForChannel(channel))`
and dispatches per pin. `makeGpioRoutingHandler` /
`makeGpioRoutingClearHandler` keep the mirror in sync. Per-board
`signalRouterMap` parallels `pinManagerMap` in lifecycle.
`makeLedcUpdateHandler` (and its memo workaround from 77bf897)
stays wired for back-compat during rollout; removed in a
follow-up commit once prod is verified stable on the new path.
Tests:
* `test/backend/unit/test_signal_router.py` (20 tests) covers
update/clear semantics, idempotency, multi-pin routing,
snapshot diff, channel↔signal-id helpers, and the multi-servo
regression scenario.
* `frontend/src/__tests__/SignalRouter.test.ts` (17 tests) is the
mirror — same scenarios on the TS side.
* `frontend/src/__tests__/esp32-multi-servo-gpio-matrix.test.ts`
(6 tests) drives the end-to-end SignalRouter handler pipeline,
asserts that two servos on GPIO 13/12 via LEDC channels 0/1
move independently (no mirroring), that re-routing carries
cleanly, and — critically — that `PinManager.broadcastPwm` is
never called.
Totals: +700 LOC, 1876 frontend tests pass (was 1853), 278 backend
unit tests pass (was 259).
Docs: ESP32_EMULATION.md §9.2 rewritten with the new architecture
diagram + a runbook for adding future peripherals through the
SignalRouter.
The C plugin hook in qemu-lcgamboa that would push gpio_out_sel
writes synchronously (eliminating the polling race window entirely)
is the next step — kept as a follow-up because the polling-fallback
path here already resolves the routing before each duty event
fires, so the bug is fixed end-to-end. The plugin work removes the
race condition fundamentally.
User-reported bug: Pi 3 simulator showed boot output but keyboard
input was ignored — the shell was effectively read-only.
Root cause: velxio-init's bash redirect was `</dev/console
>/dev/console`. From userspace, /dev/console is write-only — it is
the kernel's printk target and accepts writes (so we saw boot output
fine) but reads return EOF / block forever. Bash never saw a
keystroke and the user couldn't type.
Fix: read AND write through /dev/ttyAMA1. The 12 s devtmpfs-wait
already in velxio-init guarantees the device node exists by the
time the shell-respawn loop runs. setsid -c still gives bash a
controlling terminal so PS1, job control, and Ctrl-C all work.
Manifest version bumped to 2026-04-21+ttyAMA1; sidecar SHA check
invalidates the cached SD on every velxio backend so the fix lands
without an operator dance.
The previous velxio-init was racing devtmpfs population: its bash
redirect '</dev/ttyAMA1 >/dev/ttyAMA1' fired before the kernel had
enumerated the PL011 driver and populated the device node, so PID 1's
fd 0/1/2 redirect failed and the `while true` loop spun on
"No such file or directory" forever.
Two fixes baked into the SD image:
1. Wait up to 12 s for /dev/ttyAMA1 to appear (200 ms poll × 60).
On real bare-metal Pi the node is there at init time, but
under QEMU emulation the PL011 probe races.
2. Exec the shell with </dev/console >/dev/console — /dev/console is
set up by the kernel (no race) and points at the last `console=`
arg from the cmdline, which is ttyAMA1. Also wrap in `setsid -c`
so bash gets a controlling terminal and behaves interactively.
Verified end-to-end with a live QEMU boot against the patched .img:
shell prompt `root@raspberrypi:/#` appears within ~50 s wall (most
of that is the kernel waiting on the second SD slot mmc1 timeout
twice = 20 s).
Pi 3 simulator boot through Pi OS systemd graph was unworkable inside
QEMU's raspi3b emulation:
* The PL011 UART at 0x3f201000 enumerates as ttyAMA1 (not ttyAMA0 —
the mini-UART at 0x3f215040 takes ttyAMA0 and fails to probe under
QEMU). After ~9 s of kernel time the boot effectively went silent
on the serial: earlycon was disabled by the normal console init
and the IRQ-driven serial driver loses TX under QEMU's emulation.
* Even with `keep_bootcon`, systemd dependency graph took 2-3 min to
walk inside emulation (network waits, tmpfiles, journald,
hostname/machine-id randomness). Masking 9 boot-blocking units
helped but didn't fix the silent-after-9s problem.
Solution: skip systemd. The SD image is now baked with
`/usr/local/sbin/velxio-init` (a 30-line bash script) and the kernel
cmdline points init= at it. velxio-init mounts /proc /sys /dev /pts
/run /tmp, sets hostname, then loops a passwordless `/bin/bash
--login </dev/ttyAMA1 >/dev/ttyAMA1`. User sees the prompt within
~10 s of clicking Run; Ctrl-D respawns a fresh session.
Cmdline additions:
- `keep_bootcon` — keep earlycon alive after the regular console
registers, so kernel printk continues to reach ttyAMA1.
- `console=ttyAMA1,115200` — the correct PL011, not ttyAMA0.
- `init=/usr/local/sbin/velxio-init` — bypass systemd entirely.
Python, GPIO shim, apt, mount, etc. all work — they don't need
systemd as PID 1, just a populated rootfs + mounted pseudo-fs.
Manifest version bumped to 2026-04-21+velxio-init. Same byte size,
different SHA, so the sidecar-based cache invalidator forces a
re-fetch on every velxio backend the next time it starts.
Boot from cold to root prompt was 2-3 min because Pi OS Trixie waits
on a handful of services that timeout instead of completing:
- systemd-networkd-wait-online (60s default)
- NetworkManager-wait-online (30s default)
- wpa_supplicant + dhcpcd5 (no usable interfaces)
- raspi-config / firstboot / userconfig (no point in QEMU)
The SD image was re-baked through scripts/configure-pi3-autologin.sh
with all of them masked (the script grew a `mask_unit` helper that
symlinks each unit to /dev/null inside the rootfs). Login prompt now
appears in ~30s wall.
New manifest version 2026-04-21+autologin+fastboot — same byte count
as the previous build (still 5.4 GiB raw) but a different SHA so the
sidecar-based cache invalidation forces every container to refetch.
Two more defects making Pi 3 boot silently:
1. The kernel8.img that ships in the Pi OS armhf boot partition is a
gzip-compressed PE-COFF Image (first 4 bytes 0x1f8b0800). QEMU's
`-kernel` does NOT auto-decompress; it tries to execute the gzip
header as ARM code and the CPU faults immediately. Result: zero
bytes on ttyAMA0, simulator looks dead. Switch the asset_id to a
pre-decompressed kernel (24 MiB raw vs 9.7 MiB gzipped) so QEMU
gets a valid Image to boot.
2. Even with a real kernel, the original cmdline `console=ttyAMA0`
alone wasn't enough — the kernel can't initialise the BCM2837
PL011 UART early enough for `printk` to reach the serial console
under QEMU's bare-metal boot (no Pi firmware to set it up
beforehand). Adding `earlycon=pl011,mmio32,0x3f201000` makes the
kernel program the UART itself in the early boot path.
Verified: boot output starts streaming within 100 ms of QEMU
launch instead of never.
The cmdline also locks the baud rate at 115200 to match the agetty
drop-in created by scripts/configure-pi3-autologin.sh.
User report: clicked Pi 3 board → nothing visible happens. Three
defects, all on the same path:
1. The kernel cmdline carried over from the original pre-OSS-split
code: `quiet init=/bin/sh`. Result: kernel boot messages
suppressed, then dropped straight to bare /bin/sh with no PS1 so
the user sees an empty serial. Removed both. The kernel cmdline
is now just `console=ttyAMA0 root=/dev/mmcblk0p2 rootwait rw
dwc_otg.lpm_enable=0`, which lets systemd start a real
serial-getty@ttyAMA0.service.
2. Pi OS Trixie armhf since Bookworm ships without a default user
(no more pi/raspberry). With cmdline #1 fixed, the user would
land at a login prompt and be stuck. Fix: pre-bake a systemd
drop-in at /etc/systemd/system/serial-getty@ttyAMA0.service.d/
autologin.conf that uses `agetty --autologin root` so the serial
console drops to a root shell on first prompt. The browser
canvas IS the authentication boundary; the SD image is mounted
RO via a qcow2 overlay so per-session edits don't persist.
Edit happens in velxio-prod/scripts/configure-pi3-autologin.sh
(to follow in a separate commit).
3. Architectural: the original cache-hit probe was size-only.
Today's SD image rebake produced a file with identical byte count
but different SHA256 — the cache served stale content for every
request even after a manifest bump. Fix: write a sidecar
`<file>.sha256` after every successful materialise and trust it
on subsequent probes. Manifest SHA bumps invalidate the cache
regardless of size. Two regression tests guard this:
- test_provider_sidecar_invalidates_on_sha_mismatch
- test_provider_missing_sidecar_treats_file_as_invalid
Manifest bumped to version "2026-04-21+autologin" for the SD image
(kernel + DTB unchanged, still 2026-04-21).
Pi 3 simulation had been broken since at least April 2026 (51
fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct
defects compounded:
1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree
blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS
SD image — none of which shipped in the repo or were pulled at
image build.
2. qemu-system-arm + qemu-utils were missing from the Docker image
entirely, so even with the boot files in place QEMU couldn't
launch. Add both to Dockerfile.standalone (~200 MB).
The architecture fix is a new `app.services.boot_images` module:
* Manifest-driven (boot_images/manifest.json, versioned in repo,
declares SHA256 + size for each file, supports an optional
`compressed.{encoding,sha256,size_bytes}` block for assets shipped
as .zst).
* `BootImageProvider` materialises files lazily, atomically (temp +
rename), verifies SHA256 pre- AND post-decompression, caches under
/var/cache/velxio/boot-images, serialises concurrent get() calls
per image set via asyncio.Lock.
* `AssetDownloader` Protocol with two impls:
- `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs
use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY).
- `LocalDirectoryDownloader` — for tests + in-prod use where the
licence-module storage is already on the same filesystem (saves
the loopback HTTP roundtrip on a 1.4 GiB blob).
* `build_downloader_from_env()` picks one — local-dir wins if both
sets of env vars are present, so the prod box short-circuits to
direct disk reads automatically.
* Lifespan hook in qemu_manager.py pre-warms the cache on container
boot so first-time user requests don't pay the 30-60 s download
+ decompress latency.
Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via
upload-binary.sh, append an entry to manifest.json, register a
lifespan pre-warm in the new board's service module. Zero edits to
provider.py / downloader.py.
Manifest entries for raspberry-pi-3:
kernel8.img 9 695 883 bytes (uncompressed)
bcm2710-rpi-3-b.dtb 34 687 bytes (uncompressed)
raspios-trixie-armhf.img 5 729 419 264 bytes raw
/ 1 488 002 803 bytes .zst on wire (zstd -19)
source: 2026-04-21 build from raspberrypi.com
Tests: 21 new unit tests covering manifest parsing, integrity
helpers, both downloaders, and the provider's idempotent /
concurrent / integrity / decompression / warmup paths. In-process
FakeDownloader keeps the suite under 1 s and httpx-free.
Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk
layout, named-volume operation, and the procedure for adding a new
image set.
After Phase 4 of the OSS / pro split, the OSS code base imports zero
auth/DB modules (verified with grep across backend/app/). But the
requirements.txt + config.py + .env.example + docs still listed
SQLAlchemy, aiosqlite, JWT/bcrypt, OAuth, SECRET_KEY etc. as if they
were live. Self-hosters running `pip install -r requirements.txt`
were pulling ~30 MB of packages the code never imports.
Changes:
* backend/requirements.txt — drop sqlalchemy, greenlet, aiosqlite,
python-jose, passlib[bcrypt], bcrypt, authlib, email-validator,
python-multipart. Keep fastapi, uvicorn, websockets, pydantic,
pydantic-settings, httpx, mcp, esptool, wasmtime — everything OSS
actually uses.
* backend/app/core/config.py — Settings reduced to FRONTEND_URL only.
Comment explains the overlay path that adds the rest at Docker
build time.
* backend/.env.example — same trim: only FRONTEND_URL, with a comment
explaining why this file is almost empty.
* README.md — "Auth & Project Persistence" section rewritten to
describe .vlx export/import. Env-var table reduced to a single row.
Stack table updated: no SQLAlchemy, no JWT, persistence = .vlx
files.
* CLAUDE.md — intro line updated (Auth: None, persistence: .vlx).
Key-file-locations rewritten to list the OSS-stateless backend +
the new lib/proRoutes / proSession / proSaveAction seams, with an
explicit "removed in the split" note pointing to velxio-prod.
Stores section drops useAuthStore (overlay-only now). Backend
gotchas drop the bcrypt + email-validator + model-import notes.
Implemented-features list replaces "Auth + URL persistence + user
profile" with portable .vlx export/import.
* docs/ESP32_EMULATION.md — two `docker run` examples dropped the
`-e SECRET_KEY=...` arg (no longer needed).
OSS build verified end-to-end (285 SEO pages prerender, 20 stateless
routes, zero sqlalchemy imports).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Without the type annotation, FastAPI treats `request` as a Query
parameter and bubbles it up to every endpoint that uses
`Depends(get_current_user_id)`. Result: POST /api/compile/start
and POST /api/compile/ both returned 422
{"loc":["query","request"],"msg":"Field required"} on every call
the frontend made — compile was fully broken in production.
The frontend then caught the 422 axios error and surfaced
response.data as a CompileResult, which had no success/stdout/
stderr/error fields, so the editor's CompilationConsole rendered
only the fallback "✕ Compilation failed" line with no detail.
Annotating `request: Request` is the standard FastAPI pattern;
the framework injects the raw HTTPRequest and no longer treats
it as a query parameter.
Phase 2 of the OSS / pro split. The hook seams introduced in Phase 1
let stateless routes (compile, libraries, simulation, iot_gateway)
run without the auth/DB stack importable. Now we actually delete the
stack:
app/api/routes/auth.py
app/api/routes/projects.py
app/api/routes/admin.py
app/api/routes/metrics.py
app/models/{user,project,usage_event,password_reset_token}.py
app/schemas/{auth,admin,project}.py
app/core/{dependencies,security}.py
app/database/session.py
app/services/{metrics,odoo_mail,project_files}.py
app/utils/{geo,slug,boards}.py
Private deployments (velxio.dev) get the same modules back via the
velxio-prod overlay: pro/backend/app/api/routes/auth.py etc. are
COPYed onto /app/... at container build time, and register_pro()
includes their routers + registers the lifespan/metrics/auth hooks.
main.py shrank back to the stateless router includes + a single
`run_lifespan_startup()` call. The Phase-1 try-import block that wired
record_compile / get_current_user_id from upstream is gone — those
adapters live in pro now.
Verification:
OSS only: 20 routes (compile, libraries, simulation, gateway).
OSS + pro: 94 routes — identical to pre-refactor velxio.dev.
Net change: -2400 lines from OSS, all of which moved to velxio-prod's
overlay. Self-hosted OSS users lose accounts + project persistence;
the Phase 4 .vlx export/import gives them a portable replacement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First phase of the OSS / pro split. Goal: open the seams so the auth/DB/admin
stack can move into the private overlay (Phase 2-3) without the routes that
stay in OSS (compile, libraries, simulation, iot_gateway) having to know.
Backend
-------
* New app/core/hooks.py — registry for record_compile, get_current_user_id,
and lifespan startup tasks. Each hook is a no-op by default; overlays
call register_* in register_pro(app) to plug in a real implementation.
* compile.py now imports only from app.core.hooks. Drops the direct deps on
app.core.dependencies, app.database.session, app.models.user, and
app.services.metrics. Route signatures use `Depends(get_current_user_id)`
instead of `Depends(get_current_user)`; the metric helper passes user_id
through rather than a User instance.
* compile_chip.py drops the unused _current_user Depends entirely.
* main.py wraps the auth/DB stack import in try/except. When it succeeds
(today's behavior on velxio.dev), an adapter bridges record_compile and
get_current_user_id to the existing app.services.metrics + dependencies,
and the create_all + ALTER TABLE migration block runs via a registered
lifespan_startup hook. When it fails (the post-Phase-2 OSS image), main
logs "running stateless" and skips registering anything — the routes
still load and behave as no-ops for metrics + always-anonymous for auth.
Frontend
--------
* useAutoSaveProject becomes a skeleton: one useState + one useEffect that
delegates to an installed AutoSaveImpl. installAutoSaveImpl() replaces
the impl without changing hook count, so React's rules-of-hooks stay
satisfied even after the impl moves out of OSS.
* New hooks/autoSaveImpl.ts holds the original logic (debouncing, dirty
detection, owner eligibility, fetch keepalive on unload), refactored to
emit() instead of useState. It self-registers at module load; main.tsx
imports it for the side effect.
* AppHeader wraps the entire user-vs-login UI in a data-velxio-slot
="header-auth" boundary. Today the OSS UI still renders inside the slot
— the overlay can portal-inject additional items now, and in Phase 3
the slot becomes the sole owner of header auth UX.
Behavior is identical on velxio.dev (pro overlay imports everything
successfully, every adapter wires up). The change is purely structural:
deleting the auth/DB modules tomorrow no longer crashes OSS at import.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
arduino-esp32 v2.0.17's libraries/WiFiClientSecure/src/ssl_client.cpp:23
wraps its entire body in:
#if !defined(MBEDTLS_KEY_EXCHANGE__SOME__PSK_ENABLED) \
&& !defined(MBEDTLS_KEY_EXCHANGE_SOME_PSK_ENABLED)
# warning "Please call idf.py menuconfig ..."
#else
ssl_init / start_ssl_client / stop_ssl_socket /
send_ssl_data / get_ssl_receive / data_to_read
#endif
Our esp-idf-template/sdkconfig.defaults did not enable any PSK key-exchange
mode, so MBEDTLS_KEY_EXCHANGE_SOME_PSK_ENABLED was never auto-set by
mbedtls and ssl_client.cpp compiled to an empty translation unit. The
companion WiFiClientSecure.cpp still compiled and ended up in
libarduino-esp32.a with dangling references, breaking the link of every
sketch that pulls in HTTPClient or WiFiClientSecure (directly or
transitively).
Reproduced against the user's WiFi + HTTPClient example.com sketch on the
prod server and again locally with ESP-IDF v4.4.7 + arduino-esp32 v2.0.17;
the prebuilt sdkconfig that ships with arduino-esp32 itself sets both
flags, so we just align with that.
After the fix the same sketch links cleanly:
velxio-sketch.bin binary size 0xbf470 bytes ... 25% free
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the register-then-immediately-forgot race for good. Two parallel
calls to /velxio/api/send-welcome and /velxio/api/send-password-reset
were hitting Odoo's REPEATABLE-READ snapshot timing: both workers took
their snapshot BEFORE either had committed the partner row, so each
ran its own INSERT and the second blew up on the velxio_user_id
unique constraint — costing one of the two emails.
Add a new sync_partner() helper that POSTs to a new Odoo route
/velxio/api/upsert-partner (lives in velxio_subscription) which only
upserts the partner — no mail, no subscription, fast. Register and
forgot-password await this BEFORE firing the async welcome / reset
tasks. Each user's flow is sequential at the Velxio HTTP layer, so
the snapshot race disappears.
sync_partner() reuses the existing _post() error-swallowing pattern.
If Odoo is down, sync_partner returns None and the await is a no-op
— the user still registers / gets the generic 200, the welcome /
reset endpoints retain their defensive upsert as a fallback path.
Adds velxio_user_id to the send_password_reset payload (mirroring
send_welcome). The Odoo side's res_partner.velxio_user_id is unique,
so when Odoo eventually upserts on this endpoint the constraint
serializes concurrent register-then-immediately-forgot upserts and
prevents the duplicate-partner record the previous wire format risked.
Closes the remaining gaps in cross-board I2C so any topology of
supported boards (Uno↔ESP32, two ESP32s, Uno↔Uno↔Uno, ESP32-C3
connected to anything, etc.) works end-to-end with all I2C
components including write-only sinks (SSD1306, PCF8574, LCD-I2C).
Implementation (6 phases):
1. **BFS routing in I2CBusManager**: connectToSlave + handleExternalConnect
walk the bridge graph with a visited Set so multi-hop chains
(A↔B↔C with the device on C) resolve transparently. A new
forwarder-device shim is installed at intermediate hops so the
existing handleExternalWrite/Read/Stop machinery routes
through without per-method visited tracking.
2. **Per-peer proxy ownership in Esp32BridgeShim**: replaces the
global _proxiedAddrs Set with _proxiedByPeer Map so concurrent
bridges to the same ESP32 (e.g. wired to both Uno and Pico)
don't wipe each other's proxies on teardown. Interconnect's
per-wire teardown calls clearProxiesForPeer(peerBus) instead of
clearAllProxies.
3. **BFS-aware proxy sync**: syncProxyFromPeer now walks the peer
bus + its transitive bridges, so an ESP32 sees devices on
boards two or more hops away. _peerDeviceLookup keeps a flat
addr → device map for write-forwarding and resync.
4. **Periodic resync (250 ms)**: Esp32BridgeShim runs a setInterval
while any proxy is live, re-dumping each device with
dumpRegisters() and pushing updateProxyI2c only when an XOR-
stride hash changes. This keeps RTC time advancing visible to
ESP32 firmware without flooding the WS pipe with static
calibration dumps. Hash is primed during initial sync so the
first tick doesn't push a redundant identical buffer.
5. **Write-forwarding ProxySlave → peer**: backend ProxySlave
buffers write bytes during the transaction and emits a
`proxy_i2c_complete` event on STOP / repeated-START. Frontend
Esp32Bridge dispatches the event to a new onProxyI2cComplete
callback; the shim replays the byte sequence on the actual
peer I2CDevice via writeByte() + stop(). Makes ESP32 firmware
writes to peer SSD1306 actually repaint the OLED, peer PCF8574
latch updates, peer I2CMemoryDevice register mutations propagate.
6. **ESP32-C3 routed as bridge**: Interconnect.isBrowserSim no
longer claims c3/xiao-c3/c3-supermini — they were already
going through Esp32Bridge per the store's ESP32_RISCV_KINDS
routing, but Interconnect was treating them as browser sims
which broke proxy install. isEsp32Bridge now correctly
includes c3 family + ESP32-S3 + Arduino Nano ESP32.
Defensive: addBoard now disposes any existing shim's proxies
before overwriting simulatorMap entry so test reruns don't leak
timers.
Tests:
- 4 BFS multi-hop tests (i2c-multi-board-slave-gap.test.ts)
- 11 cross-board scenarios + per-peer + write-forward + resync
(i2c-esp32-multiboard-bridge.test.ts)
- 1 real-firmware E2E for write-forward via QEMU (compile +
load + observe proxy_i2c_complete arriving with the byte)
- New sketch fixture: esp32_i2c_write_to_peer.ino
Result: 90 test files / 1295 tests pass / 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the transactional email pipeline driven from the Odoo SMTP relay so
new sign-ups get a Velxio-branded welcome and existing users can reset a
forgotten password without us running our own outbound mail server.
Backend:
- PasswordResetToken model: one-time, SHA-256-hashed (plain text never on
disk), TTL 60 min, marked used_at on consume to prevent replay.
- POST /auth/forgot-password — anti-enumeration (always 200 + generic
message), rate-limited 3/hour/user.
- POST /auth/reset-password — verifies token, hashes new password,
atomically marks token used.
- /auth/register hooked with asyncio.create_task to fire welcome mail —
registration is never blocked on Odoo being up.
- New service app/services/odoo_mail.py: async httpx wrapper, fire-and-
forget, swallows every error so the request lifecycle stays clean.
- Settings ODOO_URL / ODOO_API_KEY / ODOO_MAIL_TIMEOUT_S /
PASSWORD_RESET_TOKEN_TTL_MINUTES / PASSWORD_RESET_RATE_LIMIT_PER_HOUR.
Frontend:
- /forgot-password page (single email field + "check your inbox" state).
- /reset-password?token=XYZ page (new password + confirmation, redirects
to /login?reset=ok on success).
- "Forgot your password?" link + green confirmation banner on /login.
- authService gains requestPasswordReset() and resetPassword().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Implemented `i2c-esp32-real-firmware.test.ts` to test ESP32 I2C communication via backend and WebSocket.
- Created `load-example-transitions.test.ts` to ensure proper loading of examples between board-less and board-based contexts.
- Added `CircuitVerificationModal.tsx` to display circuit verification results before running simulations.
- Developed `circuitVerifier.ts` to perform pre-flight checks for circuit safety, identifying potential issues like short circuits and component overloads.
- Introduced minimal ESP32 I2C master sketch `esp32_i2c_writer.ino` for testing I2C transactions.
A user reported on Discord: "the Velxio Console doesn't update anything,
it just waits until the very end and displays everything in one go".
True for the async compile path — /compile/status only carried `state`
and the final `result`, so the editor's CompilationConsole stayed empty
during the 5-7 minute cold ESP-IDF builds and dumped 1500 lines at once
when the build finished.
This wires live build output through the whole stack.
Backend (espidf_compiler.py)
- New _run_with_streaming() helper. When a progress_callback is provided
it spawns the subprocess via Popen + stdout/stderr drain threads and
invokes the callback line-by-line. When None it falls back to the
existing subprocess.run(capture_output=True) one-shot path so the
unit-test code that doesn't care about live output is unaffected.
- compile() and _compile_in_dir() take an optional ProgressCallback.
- _run_cmake / _run_ninja closures now go through _run_with_streaming
with that callback. cmake configure (~2-5 s) + ninja (~5-300+ s) both
stream now; the ninja output is the one users actually want to watch.
Backend (compile.py)
- _compile_job seeds COMPILE_JOBS[id]['stdout_buffer'] = '' and defines
on_progress_line(line) which appends to it. Buffer capped at 256 KB
(tail kept) so a runaway build can't OOM the FastAPI process.
- The buffer is preserved on both the success and the error path so
late polls still see the log even after state transitions to
done/error.
- /compile/status now returns the buffer as a `stdout` field.
CompileStatusResponse gains the field with default '' so old clients
that don't read it still work.
Frontend (compilation.ts)
- compileCode() takes a 4th argument: optional CompileProgress
callback fired every poll while state ∈ {pending, running}. Carries
the cumulative stdout (caller computes deltas) plus elapsed seconds.
- Surfaces the new `stdout` field of /compile/status and forwards it
to the callback. Errors thrown from the callback are swallowed —
a faulty UI hook must never break the polling loop.
Frontend (EditorToolbar.tsx)
- Both compileCode() call sites (Run and Compile-All) now pass an
onProgress callback. It tracks `lastStreamedLen` per-compile, splits
each new delta on newlines, and appends them as `info`-typed
CompilationLog entries via setCompileLogs. The Compile-All flow
prefixes each line with the board label so multi-board builds stay
readable.
- After the build settles, the existing parseCompileResult call still
runs and appends the structured analysis on top of the live stream
— that's where FAILED-block detection + the `error`-typed entries
that drive the auto-switch-to-errors filter live.
Net effect on the user complaint: cold ESP-IDF builds now show the
ninja [N/1483] progress lines streaming into the console as they
happen, instead of staring at an empty panel for 5-7 minutes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three coordinated fixes that together close the "ESP-IDF compile takes
5-7 min every time" gap and prevent the failure mode where a user clicking
compile multiple times spawns six ninja processes that peel each other
apart on a modest VPS.
What was wrong
- /compile/start generated a fresh uuid4 every call, so 6 clicks = 6
independent builds racing each other. Saw load average 30 on the prod
VPS during a real BMP280 attempt today.
- No concurrency limit anywhere; asyncio.create_task() fired without
gating.
- ccache was wired in last week (PR #149) but reported 18,350 cacheable
calls and **0 hits** because the build dir was a fresh
tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random
/tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags
→ different command line every compile → ccache hash miss every time.
What this PR does
1. Job deduplication (`backend/app/api/routes/compile.py`)
- New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file
names + contents + board. Order-independent.
- New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id.
- `compile_start` checks JOB_BY_KEY before spawning a new task; if a
job for this exact content is already pending or running, returns
the existing job_id (logs `[compile] dedup hit — reusing job <id>`).
- `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY,
keeping the index consistent. Edge case where two jobs share a key
(old finished, new running) is handled — only evict the key entry
if it still points at the purged job.
2. Concurrency control (`backend/app/api/routes/compile.py`)
- `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on
simultaneous compiles.
- `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so
concurrent compiles to the SAME board (sharing the persistent build
dir) serialise. Different boards still run in parallel up to the
semaphore cap.
- `_compile_job` acquires sema → per-target lock → flips state to
`running` → calls `_run_compile`. Pending state now accurately
reflects "queued waiting for resources".
3. Persistent build dir (`backend/app/services/espidf_compiler.py`)
- New `_prepare_persistent_project_dir(idf_target)` materialises
`/var/lib/velxio-build/<target>/project/` from the template on
first use; on subsequent compiles it wipes only `main/` and
`user_libs/` (the per-compile parts) and leaves `build/` alone so
ninja's incremental cache + ccache .o files survive.
- Toolchain version sentinel (`.idf_version`) wipes the whole target
dir if the ESP-IDF or arduino-esp32 version changes — cached
objects from the old toolchain are no longer ABI-compatible.
- `compile()` is now a thin dispatcher: persistent path or fallback
to the legacy `tempfile.TemporaryDirectory()` flow. The actual
build logic was extracted into `_compile_in_dir()` so both paths
share one implementation, no duplication.
- Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back
to the tempfile path without rebuilding the image. Critical for
production safety.
4. ccache normalisation (`Dockerfile.standalone`)
- + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise
absolute paths under that prefix when computing the cache key.
Robustens hits against any future subdir rearrangement.
5. Docker compose (`docker-compose.yml`)
- + named volume `velxio-build:/var/lib/velxio-build` so the persistent
build dir survives `docker compose up -d --build`.
- + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable
without rebuilding).
Expected impact
- Cold first compile per container per target: unchanged (~5-7 min).
- Same sketch re-compiled: ~2-5 s (everything cached).
- Different sketch, same target: ~5-30 s (only user code + new lib steps
rebuild; ESP-IDF base hits cache).
- Different sketch with new libraries: ~30-90 s (new lib component
compiles; rest hits cache).
- Concurrent clicks on same example: 1 build, others poll the same
job_id. No more six-ninja meltdown.
Tests
- `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability +
variance and `_purge_expired_jobs` consistency (including the
"two jobs share a key" edge case).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cold first compile per container is unchanged (cache empty). Subsequent
compiles drop from ~5-7 minutes to ~30-60 seconds because every ESP-IDF
base object (FreeRTOS, lwIP, esp_wifi, libsodium, soc, hal, …) hits the
cache. The user's BMP280 example, which hangs on cold compile, completes
near-instantly on the second attempt.
Why a transparent cache is safe: ccache hashes the preprocessed source +
flags + compiler. A cache hit only happens when the input is byte-for-byte
identical to a prior compile. Different sketches with different libraries
still get correct cache misses; there is no path where one project's
output contaminates another.
Changes
- Dockerfile.standalone: install ccache, set CCACHE_DIR=/var/cache/ccache,
IDF_CCACHE_ENABLE=1, configure 2 GB cap with compression. Compression
(level 6) cuts cache disk usage by ~40% with negligible CPU overhead.
- docker-compose.yml: named volume `ccache:/var/cache/ccache` so the
cache survives `docker compose up -d --build` (without it, every image
rebuild discards the cache).
- backend/app/services/espidf_compiler.py: pass `-DCCACHE_ENABLE=1` to
cmake when IDF_CCACHE_ENABLE is truthy. ESP-IDF's project.cmake
(`set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)` on line 374)
is what actually wires ccache in; without the cmake -D flag the env
var alone has no effect because we don't go through idf.py.
Escape hatch: set IDF_CCACHE_ENABLE=0 in compose env to disable without
rebuilding the image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The synchronous /api/compile endpoint forced one long-lived HTTP request
to span the entire build. Cloudflare's 100s edge timeout cuts that off
mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first
run). The user-visible symptom was HTTP 524 well before the backend
even noticed.
Backend (compile.py)
- New `POST /api/compile/start` returns `{job_id}` immediately and
spawns the actual compile as an asyncio.create_task background.
- New `GET /api/compile/status/{job_id}` returns the current job state
(`pending` | `running` | `done` | `error`). Each poll completes in
milliseconds, far under any edge timeout.
- Existing `POST /api/compile/` kept verbatim for backward compatibility
(AVR/RP2040 builds finish in seconds and don't trip 524).
- Build logic extracted into `_run_compile()` so both paths share one
implementation; no duplicated ESP-IDF / arduino-cli branching.
- Async path opens its own short-lived DB session via AsyncSessionLocal
for metric recording — the request-scoped session is dead by the time
the background task finishes.
- COMPILE_JOBS dict purges entries 30 minutes after completion so a
busy server doesn't grow unboundedly.
Frontend (compilation.ts)
- compileCode() now: POST /compile/start → poll /compile/status every 2s
until state ∈ {done, error}, with a 15-minute client-side cap.
- 30s axios timeout per individual call (not per build) so transient
network blips during a long compile auto-retry instead of failing.
- 404 on /status throws (job expired / server restarted); other poll
errors warn and retry. Surfaces structured error responses verbatim
so the editor's compile-error panel keeps working unchanged.
Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to
multiple FastAPI workers this needs to move to Redis or sqlite. Single-
instance is fine today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ESP32 BMP280 example compile was timing out at 98% (1473/1483 build
steps), failing with the unhelpful "ESP-IDF build timed out (300s)"
message even though every individual step was healthy.
Cold ESP-IDF builds that pull in external Arduino libraries — Adafruit
BMP280 + Adafruit BusIO + Adafruit Unified Sensor on top of the base
arduino-esp32 component tree — routinely produce ~1480 build objects.
On modest VPS hardware this takes 5-7 minutes the first time. Ninja's
incremental cache makes subsequent compiles seconds, but the first one
needs more headroom.
Constant lifted to NINJA_TIMEOUT_S so the value used in the timeout
matches the value reported in the error message — the previous code
hard-coded "300s" in two places that were free to drift apart.
Repro before: open the example "ESP32 — BMP280 Barometric Pressure"
on velxio.dev/editor on a clean container, click compile → fails after
5 minutes with timeout. After: completes in ~6 minutes on the first
run, ~5 seconds on subsequent runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI's e2e test_hcsr04_simulation.mjs caught the regression introduced by
a3f21a2 (the issue #101 fix). The worker crashes on boot with
Firmware decode error: No module named 'app'
esp32_worker.py runs as a subprocess via subprocess.Popen([sys.executable,
WORKER_PATH, ...]). When Python launches a script directly, sys.path[0]
is the SCRIPT's directory (backend/app/services/), not the backend root.
So `from app.services.esp32_flash_image import pad_to_flash_size` fails
because there is no `app/` under `backend/app/services/`.
esp32_lib_bridge.py wasn't affected because it runs in-process inside
uvicorn, where backend/ is implicitly on sys.path.
Fix: same try/except + importlib fallback the worker already uses for
esp32_i2c_slaves at the top of the file. First try the package import
(works when imported by the bridge's tests or anything else with the
backend root on sys.path), fall back to direct file loading otherwise.
Verified the fallback works in isolation by simulating the subprocess
context (sys.path containing only backend/app/services/) — the package
import fails as expected and the file-load fallback returns a properly
padded 4 MB buffer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A user reported the LEDC PWM RGB example failing to compile. The sketch
calls ledcAttach(pin, freq, resolution) — the one-shot API added in
arduino-esp32 3.x. Our toolchain image pins arduino-esp32 to 2.0.17
(matched to ESP-IDF 4.4.7 + the lcgamboa QEMU ROM), where the API is
the older two-step ledcSetup + ledcAttachPin pair. Sketches written
against 3.x docs hit "ledcAttach was not declared in this scope".
Bumping arduino-esp32 to 3.x means moving to ESP-IDF 5.x, which may
break our QEMU fork. Cheaper fix: ship a compat shim header in the
ESP-IDF project template that defines ledcAttach + ledcAttachChannel
in terms of the 2.x API, gated on `!defined(ledcAttach)` so it
disappears the day we bump.
espidf_compiler.py now injects #include "velxio_compat.h" right after
Arduino.h whether the user explicitly included Arduino.h or we
prepended it ourselves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two reports rolled into one sdkconfig change:
1. Beta testers reported "weird serial output" across DHT22, Servo+Pot,
Joystick, WiFi×2, and DualADC examples. The user's Serial.print lines
came interleaved with internal ESP-IDF chatter:
I (53306) gpio: GPIO[4]| InputEn: 1| OutputEn: 0| ...
I (10626) phy_init: phy_version 4791,2c4672b,...
sdkconfig.defaults didn't pin a default log level so it inherited
CONFIG_LOG_DEFAULT_LEVEL_INFO. Set WARN (level 2) so only warnings
and errors leak into the user's serial output. Sketches can still
esp_log_level_set() per tag at runtime if they want verbose.
2. BLE Advertise example failed to compile. sdkconfig had NimBLE
enabled, but arduino-esp32 2.0.17's BLEDevice.h targets Bluedroid.
Switch the stack: enable BT_BLUEDROID + BTDM_CTRL_MODE_BR_EDR_BLE +
BT_BLE so the standard arduino-esp32 BLE library compiles. Sketches
that explicitly include NimBLEDevice.h will not compile under this
config — that's a smaller minority than the BLEDevice.h users.
The runtime side of BLE still depends on stubs in the qemu-lcgamboa
fork; this only fixes the compile path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issue #101 reproducer: an ESP32 sketch that pulls in Adafruit_SSD1306 +
Adafruit_GFX produced a "No response from server. Is the backend
running on port 8001?" error in the browser. The compile actually
succeeded backend-side, but the JSON response carrying the firmware
was ~5.5 MB of base64 — the ESP-IDF compiler builds a full 4 MB merged
flash image (mostly 0xFF padding), encodes it whole, and ships it. In
prod that response goes through nginx + Cloudflare, which buffer-fail
or RST the connection on payloads that big — axios then lands in the
"no response" branch with no HTTP status to surface.
Fix: trim the trailing 0xFF padding before serializing, re-pad to a
valid QEMU flash size (2/4/8/16 MB) just before mtd attach. Lossless:
bytes after `last_used` in the merge are 0xFF by construction, so
trim → pad reproduces the original image byte-for-byte.
Numbers from the reproducer (Adafruit_SSD1306 + Adafruit_GFX,
esp32:esp32:esp32 board):
before: ~5.5 MB JSON response
after: 539 KB JSON response (10× smaller)
backend/app/services/espidf_compiler.py
_merge_flash_image now tracks `last_used` across the three placed
sections (bootloader / partitions / app) and writes only
flash[:last_used] to merged_flash.bin.
backend/app/services/esp32_flash_image.py (new)
Shared `pad_to_flash_size(bytes) -> bytes` helper. Rounds up to the
next valid QEMU flash size with a 4 MB minimum, matches the
frontend's existing padToFlashSize logic in Esp32MicroPythonLoader.
Raises ValueError on >16 MB inputs (would indicate a broken upstream
merge, not anything user-recoverable).
backend/app/services/esp32_lib_bridge.py
backend/app/services/esp32_worker.py
Both QEMU consumer paths (in-process and subprocess) call
pad_to_flash_size right after base64.b64decode, before writing the
tmp .bin that QEMU attaches with `-drive if=mtd,format=raw`.
Verified: smoke test confirms trim → pad → original is byte-exact.
Edge cases covered: small payloads pad up to the 4 MB minimum;
firmwares >16 MB are rejected loudly.
Closes#101
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 4 nullable columns to the users table so deployments that wire an
external billing system (e.g. velxio.dev with Odoo) have a place to cache
subscription status. Self-hosters never write these — defaults are safe
(no paid features unlock).
- models/user.py: is_paid_subscriber (bool, default false), subscription_status
(str|None), subscription_period_end (datetime|None), odoo_partner_id
(int|None, indexed).
- main.py: 4 ALTER TABLE statements appended to the legacy_migrations list
so existing deployments auto-migrate on next boot.
- schemas/auth.py: extend UserResponse with the 3 user-visible fields
(is_paid_subscriber, subscription_status, subscription_period_end). The
frontend useAuthStore already persists the whole UserResponse, so these
surface automatically without any frontend changes upstream.
odoo_partner_id stays internal — clients don't need it.
Zero behavioural change for existing OSS deployments.
Three small, backwards-compatible hooks let anyone with private features
(velxio.dev's analytics, custom integrations, paid tiers, …) layer them
on top of the open-source build without forking files.
Backend (app/main.py):
- After standard router registration, try-import an optional `app.pro`
module exposing `register_pro(app)`. ImportError is silently swallowed
(the OSS image doesn't ship `app.pro`, so this is a no-op there).
Frontend:
- EditorToolbar: new optional `rightSlot` prop renders extra elements
after the built-in right-group buttons (mirrors the existing
`centerSlot` pattern).
- main.tsx: dynamic `import('@pro/index')` gated by VITE_PRO_BUILD env.
When unset (OSS build), the branch is dead-code-eliminated and no pro
chunk is emitted.
- vite.config.ts: `@pro` alias resolves to `src/__pro_stub__/` by default.
Private builds set `VITE_PRO_BUILD=true` and `PRO_OVERLAY_PATH=<path>`
to point at their real overlay tree.
- src/__pro_stub__/index.ts: 1-line no-op `mountPro` so TypeScript and
Vite resolvers stay happy in OSS builds.
Verified: `npm run build:docker` succeeds; `npm test` passes 1161/1162;
the OSS bundle (43 MB) contains zero references to `__pro_stub__`,
`@pro`, or `pro/index` (verified via `grep dist/`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The _should_include filter inside _merge_arduino_libs_to_component
rejected every subdirectory of a library except 'utility/' when the
library lacked a src/ layout. That blocked legitimate header dirs like
Adafruit_GFX_Library/Fonts/, breaking compiles that use any GxEPD2
example with a custom font (#include <Fonts/FreeMonoBold12pt7b.h>).
The earlier filter at the top of the function already excludes
docs/examples/tests/etc. via excluded_dirs, so anything that survives
that check is presumed to be buildable source. Letting all remaining
subdirs through restores Fonts/, gfxfont/, and similar conventional
auxiliary header directories that Adafruit-style libs rely on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The transitive-include scan in _merge_arduino_libs_to_component used
glob('*.h') on the component dir, which only finds headers at the root.
Libraries with a src/ layout (GxEPD2, ArduinoJson, most modern Arduino
libs) keep their headers under src/<...>, so the scan saw zero headers
and never queued their transitive deps.
Symptom: compiling a sketch that includes GxEPD2_3C.h failed with
'Adafruit_GFX.h: No such file or directory' even though Adafruit_GFX
was installed via the Library Manager — because the BFS never reached
its header from inside GxEPD2_GFX.h.
Switching to rglob('*.h') walks the full directory tree and lets the
BFS pick up Adafruit_GFX, Adafruit_BusIO, and any other transitive
dependency that lives under src/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves several install pain points reported by users (#108, #120) and
removes the obligatory upstream-clone step that confused contributors and
slowed down every Docker build.
Install fixes:
- nginx: server_name → catch-all default_server, drop Debian's stock site
so reverse-proxied users no longer get the "Welcome to nginx" page.
- entrypoint: auto-generate SECRET_KEY at first boot, persisted under
data/.secret_key. backend/.env is now optional in docker-compose.yml.
- backend: add greenlet>=3.0.0 (SQLAlchemy async dep that was missing on
some Python builds — caused uvicorn startup failures on WSL).
Wokwi libs come from npm:
- @wokwi/elements 1.9.2, avr8js 0.21.0, rp2040js 1.3.2 are pinned in
frontend/package.json. Vite aliases removed.
- Dockerfile.standalone no longer clones avr8js / rp2040js / wokwi-elements
/ wokwi-boards. Frontend stage is just COPY + npm install + build:docker.
- Board SVGs vendored under frontend/public/boards/ (10 deduped against
existing files, 2 truly new). third-party/wokwi-* clones become reference-
only credits — generate-component-metadata.ts skips gracefully when absent.
Production config split out:
- docker-compose.prod.yml, deploy/nginx.prod.conf, nginx-host-velxio*.conf,
update-third-party.bat removed. Production deployment lives in its own
repo: https://github.com/velxio/velxio-prod (host nginx + HTTPS + backups
+ pinned upstream commit).
Verified locally: 1161 frontend tests pass, build:docker completes clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The directory grew well beyond Wokwi-only contents: it now hosts
lcgamboa's QEMU fork (qemu-lcgamboa), Espressif's esp32-camera, the
ngspice WASM build, fritzing-parts, picowi, an alternative QEMU
(qemu-esp32), the 100_Days_100_IoT_Projects examples repo, and
Wokwi's own avr8js/rp2040js/wokwi-elements/wokwi-features/wokwi-boards.
"wokwi-libs" was misleading — half the contents have nothing to do
with Wokwi. "third-party/" is the standard convention for vendored
external dependencies.
Mechanical changes:
Path rename:
wokwi-libs/ → third-party/
update-wokwi-libs.bat → update-third-party.bat
docs/WOKWI_LIBS.md → docs/THIRD_PARTY.md
Submodule reconfiguration:
.gitmodules — 4 path= and section names updated
.git/modules/wokwi-libs/ → .git/modules/third-party/
each submodule's .git file rewired to ../../.git/modules/third-party/<name>
Reference updates (~80 files): vite.config.ts aliases, Dockerfile
COPY paths, GH Actions workflow steps, build_qemu_*.sh, all
docs/* and test/*/autosearch/* entries that mention the path,
package-lock.json file: dependencies, .gitignore patterns,
sitemap.xml + index.html SEO blurbs, scripts/generate-component-*,
.dockerignore, .idea/vcs.xml. Bulk replaced both `wokwi-libs/`
(path) and bare `wokwi-libs` (textual mentions in docs/comments).
Verified:
- npx tsc -b --noEmit produces no new errors related to these paths
- vite.config.ts aliases now point at ../third-party/avr8js etc.
- All 4 git submodules (avr8js, rp2040js, wokwi-elements,
wokwi-features) are linked under third-party/ with their
worktrees re-populated and config files referencing the new path
- `grep -r wokwi-libs` returns zero hits outside node_modules,
.vite, frontend/dist, third-party/ (upstream submodule contents),
*.pyc caches, and *.dll.pre-camera rollback binaries
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User reported only 2 frames rendering after the SPI batching change.
Diagnosis: the previous flush triggers (CS-line HIGH, buffer >=4096)
were both event-driven. Adafruit_ILI9341's ESP32 backend manages CS
via digitalWrite — i.e. through the GPIO peripheral, NOT the SPI
peripheral's hardware CS pin. So the CS-line-HIGH event from
picsimlab_spi_event NEVER fires for this driver. The buffer only
flushes when it hits 4096 bytes.
Frame 1: 38 400 bytes from drawRGBBitmap → 9 flushes at 4096-byte
boundaries → last 192 bytes stay in the buffer. Status bar adds
some bytes too → maybe one more flush.
Frame 2: same. But by frame 3 the firmware is running ahead of the
flush rhythm and somehow the buffer pattern wedges in a state where
no flush completes (likely a partial buffer that sits between
transactions while the firmware briefly waits on the next fb_get).
Hard to reproduce deterministically — but the symptom matches.
Fix: add a 50 ms periodic flush thread. Independent of any event,
it acquires the lock and flushes whatever's pending. Bounds the
worst-case latency at 50 ms (= 20 fps ceiling, more than enough for
the emulator).
Triple-trigger now:
1. CS HIGH (fast path for hardware-CS drivers)
2. Buffer >= 4096 (safety for big transactions)
3. 50 ms timer (catches GPIO-CS drivers, prevents stalls)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User reported the ESP32-CAM + ILI9341 live preview at ~1 frame/min.
Profile: 80×60 preview pushes 9600 SPI bytes per drawRGBBitmap, and
each byte was emitting a full {type:'spi_event'} JSON message over
the worker→backend→WS→frontend pipeline. Per-byte overhead ~150-200µs
in Python (json.dumps + sys.stdout.write+flush dominates) plus
asyncio + WS dispatch. Net: 1.5-2 sec/frame minimum, much worse with
GIL contention.
Fix: buffer MOSI bytes in the worker and emit a single base64-encoded
`spi_batch` message when CS goes HIGH (transaction ended) or the
buffer crosses 4 KiB. ~9600 events/frame collapse to ~3 messages.
backend/app/services/esp32_worker.py:_on_spi_event
- Add _spi_byte_buf bytearray + threading.Lock
- On op==0x00 (byte): append; flush early if buf >= 4096
- On op==0x01 (CS change): flush buffer, then emit the CS event
via the legacy spi_event channel (ePaper / custom chips that
observe CS still get it).
frontend/src/simulation/Esp32Bridge.ts
- New 'spi_batch' message handler decodes b64 and replays each
byte through the existing onSpiByte callback. Parts that
subscribed via simulator.spi.onByte don't notice the protocol
change. The 'spi_event' branch still handles CS changes plus
legacy single-byte payloads for backwards compat.
Now that 38 KB/frame is cheap, restore preview to 160×120 + JPEG
quality 0.35 in the gallery example. Real measured speedup: ~50× on
the QVGA preview demo. Real hardware was never affected — it runs
SPI at 80 MHz and pushes the bitmap in ~4 ms either way.
PSRAM emulation is unrelated to this bottleneck and was left untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
David Montero Crespo