Two small fixes after running the test inside the prod container for
the first time:
- The prod image lays out the backend at /app/app/, not /app/backend/app/
(the Dockerfile.standalone COPYs only the inner package). Use /app
as the sys.path root so `from app.pro.services import ...` resolves.
- The CHIP=0x60 and BLOCK= prints race against the socket drain. The
test was treating "saw CHIP= but BLOCK= not in buffer yet" as a
hard failure and exiting before the second I2C read finished.
Gate the success path on both markers present and keep polling
otherwise.
Verified end-to-end in the prod container:
[proto] >>> ['I2C', '1', '76', 'RR', 'd0', '1']
[proto] <<< I2C_DATA 1 76 60
[proto] >>> ['I2C', '1', '76', 'RR', 'f7', '8']
[proto] <<< I2C_DATA 1 76 530280155e607b50
[test] OK — guest read chip ID = 0x60
[test] OK — block read BLOCK=530280155e607b50
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the public extension points the velxio-prod overlay uses to bind
real canvas-side I2C/SPI/UART models (BME280, future MCP23017, etc.)
to a running Pi guest's protocol shims:
- qemu_manager: set_pi_slave_handler(fn) / get_pi_slave_handler() for
pi_attach_slave + pi_detach_slave WebSocket messages. OSS image
leaves the hook unset so the messages are silently dropped.
- simulation route: parses the two new WS message types and forwards
them to the registered handler when present.
- RaspberryPi3Bridge: attachSlave(spec) / detachSlave(spec) frontend
side of the protocol.
- piSlaveScanner: at simulation start walks components + wires,
identifies I2C/SPI/UART peers wired to Pi protocol pins (40-pin
header physical-pin numbering), and emits one attach per
bus/address pair (deduped across SDA+SCL wires).
- RaspberryPiWorkspace: invokes the scanner once the bridge is open,
with retries to ride out the WS-still-connecting race.
- integration test: pi3_bme280_attach.py boots the Pi, pre-attaches a
BME280 via the slave handler, runs a host-side proto loop, runs
guest python smbus2.read_byte_data(0x76, 0xD0) and asserts the
console reads back CHIP=0x60.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Phase 2 E2E test was sending the Python GPIO command via
'python3 -c "..."' but bash quote-nesting silently corrupted the
script — the python process started, printed nothing, exited 0, and
the test asserted 'GPIO_SETUP 17 out' was missing in proto bytes
(it never got sent because the python script never ran).
Switch the test to base64-encode the script + pipe through base64 -d
into a file, then execute. Verified end-to-end now:
[test] proto received 36 bytes:
GPIO_SETUP 17 out pud_off
GPIO 17 1
[test] ✓ shim → proto pipeline works
Also bump the rootfs manifest entry to the final Phase 2 build
(d6d4a274 raw / debd1c33 zst, version 2026.05+phase2-shims-final).
Earlier auto-discovery in _transport.py was hanging at import time
on some glob/sysfs interaction. Now hardcoded /dev/vport1p1 which
is the empirical path under -M virt + virtio-blk-pci on slot 0.
raspi3b pl011 RX is broken in QEMU 10 + kernel 6.12 — see
project/pi-emulation/decisions.md for the full debugging trail.
This commit lands Phase 1 of the rebuild: switch the QEMU machine
to virt + cortex-a53, boot the velxio kernel/initramfs/rootfs over
virtio-blk-pci, and expose the user shell on /dev/hvc0 via
virtio-serial-pci + virtconsole.
End-to-end smoke verified: boot → agetty autologin → bash prompt →
echo round-trip returns the typed token. Tested inside the prod
container with QEMU 10.0.8 and our cloud-derived kernel 6.12.88.
What changed:
backend/app/services/qemu_manager.py
PI3_IMAGE_SET -> raspberry-pi-3-virt
PI3_KERNEL_NAME / PI3_INITRAMFS_NAME / PI3_ROOTFS_NAME new
QEMU cmd rewritten end-to-end:
-M virt -cpu cortex-a53 -smp 4 -m 1G
-kernel <velxio-kernel-arm64> -initrd <velxio-initramfs-arm64.cpio.gz>
-drive ... -device virtio-blk-pci (NOT virtio-blk-device — mmio
variant left /dev/vda unregistered)
-nic none -display none -monitor none -serial none
-chardev socket... -device virtio-serial-pci -device virtconsole
(user console -> /dev/hvc0)
-chardev socket... -device virtserialport,name=velxio-protocol
(Phase 2 channel -> /dev/vport0p2)
No -dtb (virt generates its own), no -append init=... (kernel runs
our initramfs which then switch_root to rootfs and exec's its
/sbin/init — Alpine OpenRC).
backend/app/services/boot_images/manifest.json
New image set raspberry-pi-3-virt with three assets uploaded via
the existing license-endpoint pipeline. Old raspberry-pi-3 entry
flagged deprecated:true and kept for one release for rollback.
test/pi3_console_boot/test_pi3_console_boot.py
Updated QEMU argv to match qemu_manager exactly. Markers now look
for the Velxio Pi Simulator MOTD + 'login on hvc0' (autologin
proof). Round-trip echo still required to pass.
test_busy_wait_100us and test_busy_wait_1us measure busy_wait_us()
elapsed time against absolute thresholds (500µs / 100µs). Under
contended CI/deploy-gate machines these can blow through the budget
even when the busy-wait implementation is correct, blocking deploys
that have nothing to do with DHT22 timing.
Same pattern already applied to test_response_timing_analysis in
this file — skipped via @unittest.skipIf(os.environ['CI']=='true').
User report: on the solar-tracker project (5218f9e3) only one servo
moved and the log showed `ch=0 duty=X% gpio=12` (wrong — servoPan was
attached to GPIO 13) and `ch=1 ... gpio=-1` (servoTilt's channel
never resolved).
Root cause traced through the GPIO Matrix dump: the firmware does
exactly what the Arduino-ESP32 Servo library says — `ledcAttachPin(
13, 0)` writes signal 71 (LEDC_HS_SIG_OUT0) into `gpio_out_sel[13]`,
and `ledcAttachPin(12, 1)` writes signal 72 (LEDC_HS_SIG_OUT1) into
`gpio_out_sel[12]`. Per the ESP32 Technical Reference Manual section
4.11, Table 4-3:
71 .. 78 → LEDC HS channels 0..7
79 .. 86 → LEDC LS channels 0..7
The legacy worker code at esp32_worker.py:426 used the off-by-one
range `72 <= signal <= 87` with `ledc_ch = signal - 72`. The mistake
masked itself for single-servo projects because the 0x5000 duty
callback's channel index was internally consistent with the bogus
math, so the duty STILL reached the correctly-routed pin (just
labelled wrong). The new SignalRouter unit tests caught the
discrepancy the moment two servos drove distinct channels: signal
71 (HS_CH0, gpio 13) was REJECTED by the off-by-one filter and
signal 72 (HS_CH1, gpio 12) was misclassified as channel 0.
When I ported the legacy range into `esp32_signals.SIG_LEDC_HS_CH0_OUT_IDX`
the bug came along for the ride. Fix both modules:
* `backend/app/services/esp32_signals.py`: HS 71-78, LS 79-86.
* `frontend/src/simulation/esp32-signals.ts`: mirror.
* tests updated; 20 backend + 23 frontend pass.
After deploy the user's two servos will resolve to their declared
pins:
ch=0 duty=X% gpio=13 (servoPan, was wrongly emitting gpio=12)
ch=1 duty=X% gpio=12 (servoTilt, was wrongly emitting gpio=-1)
This is also why the multi-servo blink "patch" in commit 77bf897
appeared to help: with both pins ALIASED to the same channel via
the off-by-one, the broadcast fallback was the only thing producing
ANY movement on the second servo at all.
Replaces the per-peripheral ad-hoc `_ledc_gpio_map` cache with a
proper signal-routing abstraction that mirrors the ESP32 SoC's
IO_MUX + GPIO Matrix exactly. Same idea as real silicon: signal
sources (LEDC channels, RMT, MCPWM, ...) → 40-entry routing table
→ GPIO pins.
Motivation (from user bug report in
velxio.dev/project/5218f9e3-136d-43b3-bba1-6cebde21e1a4): two
ESP32 servos on a solar-tracker visibly oscillated between two
positions instead of moving smoothly when the user changed LDR
sliders. Commit 77bf897 patched it (per-channel gpio memo +
broadcast guard) but the user requested a proper hardware-fidel
architecture, not patches.
Backend:
* `app/services/signal_router.py` — SignalRouter class. Forward
index (gpio → signal_id) + reverse index (signal_id → set of
gpios). `replace_snapshot()` returns the diff for the polling-
fallback path; future C plugin hook becomes a push without
touching this code.
* `app/services/esp32_signals.py` — Signal id constants from
ESP32 TRM (LEDC HS 72-79, LS 80-87) + `ledc_signal_for_channel()`
helper.
* `app/services/esp32_worker.py` — `_ledc_gpio_map` is gone;
`_refresh_ledc_gpio_map` replaced by `_refresh_signal_routing`
which emits `gpio_routing {gpio, signal_id}` events on diff.
The 0x5000 LEDC callback and the LEDC poll thread now emit
`ledc_duty {channel, duty_pct}` (canonical, no gpio) alongside
the legacy `ledc_update {channel, duty, gpio}` for back-compat
during rollout.
Frontend:
* `simulation/SignalRouter.ts` — 1-to-1 TS mirror of the Python
class. Same forward + reverse index; same `pinsForSignal` /
`updateRouting` / `clearRouting` API.
* `simulation/esp32-signals.ts` — Signal id constants, mirror
of the Python module.
* `simulation/Esp32Bridge.ts` — new `onLedcDuty`, `onGpioRouting`,
`onGpioRoutingClear` callbacks; handlers for the new event types.
* `store/useSimulatorStore.ts` — `makeLedcDutyHandler` looks up
pins via `router.pinsForSignal(ledcSignalForChannel(channel))`
and dispatches per pin. `makeGpioRoutingHandler` /
`makeGpioRoutingClearHandler` keep the mirror in sync. Per-board
`signalRouterMap` parallels `pinManagerMap` in lifecycle.
`makeLedcUpdateHandler` (and its memo workaround from 77bf897)
stays wired for back-compat during rollout; removed in a
follow-up commit once prod is verified stable on the new path.
Tests:
* `test/backend/unit/test_signal_router.py` (20 tests) covers
update/clear semantics, idempotency, multi-pin routing,
snapshot diff, channel↔signal-id helpers, and the multi-servo
regression scenario.
* `frontend/src/__tests__/SignalRouter.test.ts` (17 tests) is the
mirror — same scenarios on the TS side.
* `frontend/src/__tests__/esp32-multi-servo-gpio-matrix.test.ts`
(6 tests) drives the end-to-end SignalRouter handler pipeline,
asserts that two servos on GPIO 13/12 via LEDC channels 0/1
move independently (no mirroring), that re-routing carries
cleanly, and — critically — that `PinManager.broadcastPwm` is
never called.
Totals: +700 LOC, 1876 frontend tests pass (was 1853), 278 backend
unit tests pass (was 259).
Docs: ESP32_EMULATION.md §9.2 rewritten with the new architecture
diagram + a runbook for adding future peripherals through the
SignalRouter.
The C plugin hook in qemu-lcgamboa that would push gpio_out_sel
writes synchronously (eliminating the polling race window entirely)
is the next step — kept as a follow-up because the polling-fallback
path here already resolves the routing before each duty event
fires, so the bug is fixed end-to-end. The plugin work removes the
race condition fundamentally.
test_response_timing_analysis measures the actual µs duration of the
DHT22 preamble LOW pulse and asserts it stays under 1000µs (real
hardware target ~80µs, busy-wait tolerance ~500µs). On a deploy box
under load (concurrent docker build + zstd compression + container
runtime) GIL contention inflates the observed timing far past the
threshold — the deploy gate just hit 3242µs and aborted.
Mirror the same @skipIf(CI=='true') gate the sibling
test_response_data_matches_payload already has (line 390-393).
Locally / when debugging the DHT22 path the test still runs in full.
deploy.sh's vitest + pytest output was polluted with three benign
but loud warnings that buried real signal:
1. AVRSimulator.start() unconditionally read `window.__spiceDebug`.
In node-side vitest runs `window` is undefined → ReferenceError
→ console.warn('[spice] debug dump failed', e). Logged once per
AVR test. Guarded with `typeof window !== 'undefined'`; in
production the browser path is unchanged.
2. pinPositionCalculator.calculatePinPosition() warned every time
document.getElementById returned null. In node-side tests there
is no real DOM and every wire-related test triggers the warning
for every component. Skip the console.warn when
import.meta.env.MODE === 'test' (vitest sets MODE=test); the
function still returns null and production retains the
actionable warning for unmounted components.
3. test_esp32_wifi_args.py::test_start_instance_accepts_wifi_params
mocked asyncio.create_task with no side_effect, so the coroutine
from self._boot(...) leaked and triggered a "coroutine never
awaited" RuntimeWarning. Mock now closes the coroutine.
After fixes:
frontend tests: 0 spice/pinPositionCalculator stderr lines
backend tests: 259 passed, 15 skipped, 1 warning (starlette
third-party python_multipart deprecation —
not ours, fixed when starlette updates).
User report: clicked Pi 3 board → nothing visible happens. Three
defects, all on the same path:
1. The kernel cmdline carried over from the original pre-OSS-split
code: `quiet init=/bin/sh`. Result: kernel boot messages
suppressed, then dropped straight to bare /bin/sh with no PS1 so
the user sees an empty serial. Removed both. The kernel cmdline
is now just `console=ttyAMA0 root=/dev/mmcblk0p2 rootwait rw
dwc_otg.lpm_enable=0`, which lets systemd start a real
serial-getty@ttyAMA0.service.
2. Pi OS Trixie armhf since Bookworm ships without a default user
(no more pi/raspberry). With cmdline #1 fixed, the user would
land at a login prompt and be stuck. Fix: pre-bake a systemd
drop-in at /etc/systemd/system/serial-getty@ttyAMA0.service.d/
autologin.conf that uses `agetty --autologin root` so the serial
console drops to a root shell on first prompt. The browser
canvas IS the authentication boundary; the SD image is mounted
RO via a qcow2 overlay so per-session edits don't persist.
Edit happens in velxio-prod/scripts/configure-pi3-autologin.sh
(to follow in a separate commit).
3. Architectural: the original cache-hit probe was size-only.
Today's SD image rebake produced a file with identical byte count
but different SHA256 — the cache served stale content for every
request even after a manifest bump. Fix: write a sidecar
`<file>.sha256` after every successful materialise and trust it
on subsequent probes. Manifest SHA bumps invalidate the cache
regardless of size. Two regression tests guard this:
- test_provider_sidecar_invalidates_on_sha_mismatch
- test_provider_missing_sidecar_treats_file_as_invalid
Manifest bumped to version "2026-04-21+autologin" for the SD image
(kernel + DTB unchanged, still 2026-04-21).
rp2040js runs at ~50% real time, so a TFT frame burst (fillRect sky +
fillRect floor + many drawFastVLine for walls + HUD) often takes longer
than 16 ms to drain through the SPI pipeline. Painting on every rAF
captured mid-burst snapshots that the next sky fill immediately
clobbered, so the canvas only ever showed the last few pixels written
before each tick — most visibly the raycaster examples rendering 2-3
wall columns instead of 160.
Strategy: each SPI pixel write resets a 16 ms idle timer. We paint only
after that period of silence (a real frame boundary), with a 100 ms
hard cap so continuous-write sketches still update.
Also adds test/pico_doom_demo/raycaster-perf.mjs — a puppeteer-based
profiler that reports CPU step rate, SPI throughput, per-pixel cost,
and paint rate. Run with the dev backend + frontend up:
node test/pico_doom_demo/raycaster-perf.mjs
After the fix the Doom raycaster paints at the sketch's natural 10 FPS
with full frames (was 29 fps of mid-burst snapshots).
Pi 3 simulation had been broken since at least April 2026 (51
fail-events / 24h per docs/PI3_EMULATION_BROKEN.md). Two distinct
defects compounded:
1. qemu_manager.py hard-coded paths for kernel8.img, a device-tree
blob (under a DOS 8.3 short name!), and a 5.4 GiB Raspberry Pi OS
SD image — none of which shipped in the repo or were pulled at
image build.
2. qemu-system-arm + qemu-utils were missing from the Docker image
entirely, so even with the boot files in place QEMU couldn't
launch. Add both to Dockerfile.standalone (~200 MB).
The architecture fix is a new `app.services.boot_images` module:
* Manifest-driven (boot_images/manifest.json, versioned in repo,
declares SHA256 + size for each file, supports an optional
`compressed.{encoding,sha256,size_bytes}` block for assets shipped
as .zst).
* `BootImageProvider` materialises files lazily, atomically (temp +
rename), verifies SHA256 pre- AND post-decompression, caches under
/var/cache/velxio/boot-images, serialises concurrent get() calls
per image set via asyncio.Lock.
* `AssetDownloader` Protocol with two impls:
- `LicenseGatedDownloader` — same flow ESP32 / RISC-V QEMU libs
use (VELXIO_BINARY_BASE_URL + VELXIO_LICENSE_KEY).
- `LocalDirectoryDownloader` — for tests + in-prod use where the
licence-module storage is already on the same filesystem (saves
the loopback HTTP roundtrip on a 1.4 GiB blob).
* `build_downloader_from_env()` picks one — local-dir wins if both
sets of env vars are present, so the prod box short-circuits to
direct disk reads automatically.
* Lifespan hook in qemu_manager.py pre-warms the cache on container
boot so first-time user requests don't pay the 30-60 s download
+ decompress latency.
Adding a future board kind (Pi 4 / Pi 5) is now: upload assets via
upload-binary.sh, append an entry to manifest.json, register a
lifespan pre-warm in the new board's service module. Zero edits to
provider.py / downloader.py.
Manifest entries for raspberry-pi-3:
kernel8.img 9 695 883 bytes (uncompressed)
bcm2710-rpi-3-b.dtb 34 687 bytes (uncompressed)
raspios-trixie-armhf.img 5 729 419 264 bytes raw
/ 1 488 002 803 bytes .zst on wire (zstd -19)
source: 2026-04-21 build from raspberrypi.com
Tests: 21 new unit tests covering manifest parsing, integrity
helpers, both downloaders, and the provider's idempotent /
concurrent / integrity / decompression / warmup paths. In-process
FakeDownloader keeps the suite under 1 s and httpx-free.
Docs: new docs/BOOT_IMAGES.md describes the architecture, on-disk
layout, named-volume operation, and the procedure for adding a new
image set.
A new entry in the games category for the Raspberry Pi Pico. Renders
a first-person 3D corridor à la Wolfenstein / early Doom using a
column-wise DDA raycaster — 160 rays per frame drawn straight to a
320×240 ILI9341 TFT via drawFastVLine, no framebuffer. 16×16 tile map
with 5 wall palettes (slate, blood, brown, toxic green, bronze door),
darkened on NS faces so corners read in 3D. Forward / back move,
two more buttons turn the player. 40-px HUD bar.
Why a demo, not the canonical id Software Doom: Graham Sanderson's
rp2040-doom port shoehorns DOOM1.WAD into 2 MB of flash with custom
compression and pushes video out over PIO-driven DVI / VGA — none of
that survives the rp2040js emulator (no PIO accuracy, no flash
mapping for huge assets). A raycaster reproduces the *visual* of
early Doom using only ~67 KB of flash and 9 KB of RAM, which the
emulator runs perfectly.
Pre-flight: arduino-cli compile against rp2040:rp2040:rpipico
already verified inside the Velxio backend container — 3 % flash,
3 % RAM. The Adafruit_GFX + Adafruit_ILI9341 libs the example
declares are already in the gallery's auto-install list.
Bundled:
test/pico_doom_demo/arduino_sketch.ino — source of truth sketch
test/pico_doom_demo/README.md — what + why + pin map
test/pico_doom_demo/compile_check.sh — operator script that
runs arduino-cli against the same FQBN the prod backend uses
frontend/src/data/examples.ts — gallery entry (boardType
raspberry-pi-pico, category games, difficulty advanced, 5
components, 14 wires)
frontend/src/__tests__/examples-pico-doom.test.ts — 10 vitest
assertions: example is registered exactly once, target board /
category / difficulty match, libraries declared, all four
pushbuttons present, every wire endpoint references a real
component id, SPI pin mapping matches the sketch's #define
block, every button has a GND wire, the renderFrame loop is
still present in the embedded code.
Closes the remaining gaps in cross-board I2C so any topology of
supported boards (Uno↔ESP32, two ESP32s, Uno↔Uno↔Uno, ESP32-C3
connected to anything, etc.) works end-to-end with all I2C
components including write-only sinks (SSD1306, PCF8574, LCD-I2C).
Implementation (6 phases):
1. **BFS routing in I2CBusManager**: connectToSlave + handleExternalConnect
walk the bridge graph with a visited Set so multi-hop chains
(A↔B↔C with the device on C) resolve transparently. A new
forwarder-device shim is installed at intermediate hops so the
existing handleExternalWrite/Read/Stop machinery routes
through without per-method visited tracking.
2. **Per-peer proxy ownership in Esp32BridgeShim**: replaces the
global _proxiedAddrs Set with _proxiedByPeer Map so concurrent
bridges to the same ESP32 (e.g. wired to both Uno and Pico)
don't wipe each other's proxies on teardown. Interconnect's
per-wire teardown calls clearProxiesForPeer(peerBus) instead of
clearAllProxies.
3. **BFS-aware proxy sync**: syncProxyFromPeer now walks the peer
bus + its transitive bridges, so an ESP32 sees devices on
boards two or more hops away. _peerDeviceLookup keeps a flat
addr → device map for write-forwarding and resync.
4. **Periodic resync (250 ms)**: Esp32BridgeShim runs a setInterval
while any proxy is live, re-dumping each device with
dumpRegisters() and pushing updateProxyI2c only when an XOR-
stride hash changes. This keeps RTC time advancing visible to
ESP32 firmware without flooding the WS pipe with static
calibration dumps. Hash is primed during initial sync so the
first tick doesn't push a redundant identical buffer.
5. **Write-forwarding ProxySlave → peer**: backend ProxySlave
buffers write bytes during the transaction and emits a
`proxy_i2c_complete` event on STOP / repeated-START. Frontend
Esp32Bridge dispatches the event to a new onProxyI2cComplete
callback; the shim replays the byte sequence on the actual
peer I2CDevice via writeByte() + stop(). Makes ESP32 firmware
writes to peer SSD1306 actually repaint the OLED, peer PCF8574
latch updates, peer I2CMemoryDevice register mutations propagate.
6. **ESP32-C3 routed as bridge**: Interconnect.isBrowserSim no
longer claims c3/xiao-c3/c3-supermini — they were already
going through Esp32Bridge per the store's ESP32_RISCV_KINDS
routing, but Interconnect was treating them as browser sims
which broke proxy install. isEsp32Bridge now correctly
includes c3 family + ESP32-S3 + Arduino Nano ESP32.
Defensive: addBoard now disposes any existing shim's proxies
before overwriting simulatorMap entry so test reruns don't leak
timers.
Tests:
- 4 BFS multi-hop tests (i2c-multi-board-slave-gap.test.ts)
- 11 cross-board scenarios + per-peer + write-forward + resync
(i2c-esp32-multiboard-bridge.test.ts)
- 1 real-firmware E2E for write-forward via QEMU (compile +
load + observe proxy_i2c_complete arriving with the byte)
- New sketch fixture: esp32_i2c_write_to_peer.ino
Result: 90 test files / 1295 tests pass / 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Implemented `i2c-esp32-real-firmware.test.ts` to test ESP32 I2C communication via backend and WebSocket.
- Created `load-example-transitions.test.ts` to ensure proper loading of examples between board-less and board-based contexts.
- Added `CircuitVerificationModal.tsx` to display circuit verification results before running simulations.
- Developed `circuitVerifier.ts` to perform pre-flight checks for circuit safety, identifying potential issues like short circuits and component overloads.
- Introduced minimal ESP32 I2C master sketch `esp32_i2c_writer.ino` for testing I2C transactions.
- Implement HD44780Decoder for decoding I2C commands to HD44780-compatible LCDs.
- Add bmp280_bridge_reader.ino to read BMP280 chip_id and status registers via I2C.
- Create i2c_scanner_multi.ino to scan I2C addresses and report responding devices.
- Introduce lcd_i2c_hello.ino to demonstrate basic LCD functionality with I2C.
- Implement pcf8574_bidirectional.ino to test bidirectional communication with PCF8574.
- Add pico_i2c_master_reader.ino for reading BMP280 from a Raspberry Pi Pico.
- Create rtc_lcd_clock.ino to display time from a DS1307 RTC on an I2C LCD.
Three coordinated fixes that together close the "ESP-IDF compile takes
5-7 min every time" gap and prevent the failure mode where a user clicking
compile multiple times spawns six ninja processes that peel each other
apart on a modest VPS.
What was wrong
- /compile/start generated a fresh uuid4 every call, so 6 clicks = 6
independent builds racing each other. Saw load average 30 on the prod
VPS during a real BMP280 attempt today.
- No concurrency limit anywhere; asyncio.create_task() fired without
gating.
- ccache was wired in last week (PR #149) but reported 18,350 cacheable
calls and **0 hits** because the build dir was a fresh
tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random
/tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags
→ different command line every compile → ccache hash miss every time.
What this PR does
1. Job deduplication (`backend/app/api/routes/compile.py`)
- New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file
names + contents + board. Order-independent.
- New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id.
- `compile_start` checks JOB_BY_KEY before spawning a new task; if a
job for this exact content is already pending or running, returns
the existing job_id (logs `[compile] dedup hit — reusing job <id>`).
- `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY,
keeping the index consistent. Edge case where two jobs share a key
(old finished, new running) is handled — only evict the key entry
if it still points at the purged job.
2. Concurrency control (`backend/app/api/routes/compile.py`)
- `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on
simultaneous compiles.
- `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so
concurrent compiles to the SAME board (sharing the persistent build
dir) serialise. Different boards still run in parallel up to the
semaphore cap.
- `_compile_job` acquires sema → per-target lock → flips state to
`running` → calls `_run_compile`. Pending state now accurately
reflects "queued waiting for resources".
3. Persistent build dir (`backend/app/services/espidf_compiler.py`)
- New `_prepare_persistent_project_dir(idf_target)` materialises
`/var/lib/velxio-build/<target>/project/` from the template on
first use; on subsequent compiles it wipes only `main/` and
`user_libs/` (the per-compile parts) and leaves `build/` alone so
ninja's incremental cache + ccache .o files survive.
- Toolchain version sentinel (`.idf_version`) wipes the whole target
dir if the ESP-IDF or arduino-esp32 version changes — cached
objects from the old toolchain are no longer ABI-compatible.
- `compile()` is now a thin dispatcher: persistent path or fallback
to the legacy `tempfile.TemporaryDirectory()` flow. The actual
build logic was extracted into `_compile_in_dir()` so both paths
share one implementation, no duplication.
- Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back
to the tempfile path without rebuilding the image. Critical for
production safety.
4. ccache normalisation (`Dockerfile.standalone`)
- + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise
absolute paths under that prefix when computing the cache key.
Robustens hits against any future subdir rearrangement.
5. Docker compose (`docker-compose.yml`)
- + named volume `velxio-build:/var/lib/velxio-build` so the persistent
build dir survives `docker compose up -d --build`.
- + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable
without rebuilding).
Expected impact
- Cold first compile per container per target: unchanged (~5-7 min).
- Same sketch re-compiled: ~2-5 s (everything cached).
- Different sketch, same target: ~5-30 s (only user code + new lib steps
rebuild; ESP-IDF base hits cache).
- Different sketch with new libraries: ~30-90 s (new lib component
compiles; rest hits cache).
- Concurrent clicks on same example: 1 build, others poll the same
job_id. No more six-ninja meltdown.
Tests
- `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability +
variance and `_purge_expired_jobs` consistency (including the
"two jobs share a key" edge case).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the loop on the four ESP32 fixes that landed earlier in this
series. Each previously-broken or noisy example now has a regression
test that compiles the sketch through the production ESP-IDF compiler
and runs it in QEMU via esp32_worker.py — the same code path the
WebSocket /ws/{client_id} endpoint drives in production. Testing the
worker directly skips the WS transport but exercises the same compile
→ flash → boot → serial cascade.
Coverage:
- TestEsp32SerialCleanliness — DHT22, Servo+Pot, Joystick, Dual ADC
Asserts the user's Serial.print substring shows up AND no
`I (xxx) gpio:|wifi:|phy:` info-level ESP-IDF logs leak through.
This validates the sdkconfig CONFIG_LOG_DEFAULT_LEVEL_WARN change
from commit b373c97.
- TestEsp32CompileSuccess — BLE Advertise, LEDC RGB
BLE Advertise validates the sdkconfig switch to Bluedroid (was
NimBLE-only, which broke arduino-esp32's BLEDevice.h).
LEDC RGB validates velxio_compat.h's ledcAttach() shim from
commit f6f6f43; the sketch uses the arduino-esp32 3.x one-shot API
on a 2.0.17 toolchain.
- TestEsp32WiFiSketches — WiFi Connect, WiFi WebServer
Regression coverage to make sure the sdkconfig changes didn't break
WiFi association. Connect must reach an "IP Address:" line; Server
must report "Server started".
- frontend/src/__tests__/component-metadata-bmp280.test.ts
Sanity check that the BMP280 entry from commit 1f3f2e0 survives
metadata regeneration.
All ESP-IDF/QEMU tests use unittest.skipUnless on
_toolchain_available() so they no-op cleanly on dev boxes without
libqemu-xtensa, and only do real work in the Docker CI image.
Sketches are inlined verbatim from the public Velxio examples.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Investigation into adding ESP32-P4 to Velxio via either of the two existing
emulation paths (frontend JS/WASM or backend QEMU/WebSocket). Verified the
arduino-cli toolchain works (RISC-V 32-bit ELF, RVC, single-float ABI), but
both emulation paths are blocked upstream:
- espressif/qemu has no esp32p4 machine yet (issue #127, status: To Do).
- No open-source JS/WASM ESP32 emulator exists; Wokwi's engine is closed.
Includes a smoke-test script ready for the day the Espressif QEMU machine
lands, plus a Phase A/B/C plan in autosearch/06_recommendations.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1. Backend test (test_arduino_cli_attinycore.py): the entrypoint script
was renamed deploy/ → docker/ in commit b736aea but this test still
pointed at the old path. Update the read_text() call + docstring.
2. Frontend CI (frontend-tests.yml): the cache key
`frontend-${{ hashFiles('frontend/package-lock.json') }}` was tied to
a file that has since been gitignored (commit eb9a3ec). hashFiles()
on a missing file returns the same empty hash forever, so every CI
run was restoring the same stale node_modules — including the
symlinks to `file:../third-party/wokwi-elements` that existed before
the npm migration in commit 531c337. On revalidation, npm tried to
run wokwi-elements' `prepare` script (`husky install && npm run
build`), which failed with "husky: not found".
Drop the cache step entirely; lock files aren't committed so cache
keys can't be made meaningful without overcomplication. Adds ~30s
per CI run, but actually correct. Also pass --no-audit --no-fund
to npm install for cleaner logs.
The directory grew well beyond Wokwi-only contents: it now hosts
lcgamboa's QEMU fork (qemu-lcgamboa), Espressif's esp32-camera, the
ngspice WASM build, fritzing-parts, picowi, an alternative QEMU
(qemu-esp32), the 100_Days_100_IoT_Projects examples repo, and
Wokwi's own avr8js/rp2040js/wokwi-elements/wokwi-features/wokwi-boards.
"wokwi-libs" was misleading — half the contents have nothing to do
with Wokwi. "third-party/" is the standard convention for vendored
external dependencies.
Mechanical changes:
Path rename:
wokwi-libs/ → third-party/
update-wokwi-libs.bat → update-third-party.bat
docs/WOKWI_LIBS.md → docs/THIRD_PARTY.md
Submodule reconfiguration:
.gitmodules — 4 path= and section names updated
.git/modules/wokwi-libs/ → .git/modules/third-party/
each submodule's .git file rewired to ../../.git/modules/third-party/<name>
Reference updates (~80 files): vite.config.ts aliases, Dockerfile
COPY paths, GH Actions workflow steps, build_qemu_*.sh, all
docs/* and test/*/autosearch/* entries that mention the path,
package-lock.json file: dependencies, .gitignore patterns,
sitemap.xml + index.html SEO blurbs, scripts/generate-component-*,
.dockerignore, .idea/vcs.xml. Bulk replaced both `wokwi-libs/`
(path) and bare `wokwi-libs` (textual mentions in docs/comments).
Verified:
- npx tsc -b --noEmit produces no new errors related to these paths
- vite.config.ts aliases now point at ../third-party/avr8js etc.
- All 4 git submodules (avr8js, rp2040js, wokwi-elements,
wokwi-features) are linked under third-party/ with their
worktrees re-populated and config files referencing the new path
- `grep -r wokwi-libs` returns zero hits outside node_modules,
.vite, frontend/dist, third-party/ (upstream submodule contents),
*.pyc caches, and *.dll.pre-camera rollback binaries
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User goal: ESP32-CAM live preview that works with any webcam,
regardless of resolution, brand, or scene complexity. The previous
fixed-quality 0.28 was fragile (intermittent decode errors on
moving/textured scenes) and capped visual quality unnecessarily.
Two-layer fix; either alone is insufficient:
LAYER A — Bounded JPEG encoder (frontend, this repo)
frontend/src/hooks/useWebcamFrames.ts:
encodeBoundedJpeg() walks a quality ladder [0.6, 0.5, ..., 0.1]
until the JPEG fits in MAX_FRAME_BYTES (23 000). If even q=0.1
overshoots — extreme HD/4K scenes — falls back to a 240×180
downscaled canvas at q=0.4. Guarantees every emitted frame fits
the deliverable budget regardless of webcam hardware.
The hook now exposes lastQualityUsed + lastDownscaled so UI can
surface when auto-tuning kicks in.
frontend/src/components/simulator/CameraToggle.tsx:
Tooltip shows "(auto-tuned to q=0.X)" or "(auto-downscaled, q=0.X)"
while streaming so users see what the encoder picked.
LAYER B — Multi-lap descriptor ring walker (qemu-lcgamboa, submodule)
Bumps the QEMU per-frame deliverable cap from 8 KiB to ~32 KiB by
letting the walker reset the descriptor ring up to 4 times per
VSYNC. Submodule pointer bumped to eb8b7a5d.
Combined, the demo now supports:
- Cheap 480p webcams: q=0.6, 5-10 KiB JPEGs, sharp
- Logitech mid-range: q=0.5-0.6, 8-15 KiB JPEGs, sharp
- HD 1080p webcams: q=0.4-0.6, 15-23 KiB JPEGs, sharp
- 4K complex scenes: downscaled, still readable
Documented as bug closure in:
test/test-esp32-cam/autosearch/15_universal_webcam_compat.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User confirmed the FULL pipeline works with their laptop webcam at
QVGA quality 0.6 — frontend → WS → backend → worker → DLL → I²S →
firmware → esp_camera_fb_get() → user sketch.
[Frame #1] 8192 bytes 320x240 fmt=4
├─ SOI (FF D8 FF): ✓ at offset 0
├─ EOI (FF D9): ✓ at offset 8190
└─ First 16 bytes: FF D8 FF E0 00 10 4A 46 49 46 …
(JFIF, real webcam)
Stats: 10/10 Valid JPEGs at ~3.5 fps.
Changes:
- wokwi-libs/qemu-lcgamboa @ e4321d1 (picsimlab-esp32):
EOFS_PER_FRAME 6→8, inject_eoi_now flag for EOI injection on the
last EOF of each VSYNC burst. Handles JPEGs of arbitrary size by
forcing FF D9 at offset 8190 — JPEG decoders tolerate the
truncation gracefully.
- backend/esp32_worker.py: throttled trace log every 30 frames
(`camera_frame #N received (NNNN bytes)`) so users can confirm
the frontend → worker leg is alive without flooding the log.
- test/test-esp32-cam/autosearch/14: documented bug #9 (the 9th and
final silent bug — real webcam JPEGs exceed the deliverable byte
budget) with full forensic trace + final architecture diagram.
The emulation now handles ANY user webcam → ESP32-CAM use case end
to end. Standard upstream esp_camera_init() / esp_camera_fb_get()
sketches work without modification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
End-to-end ESP32-CAM emulation now works. esp_camera_fb_get() in
user sketches returns valid camera_fb_t* pointers with JPEG frames
sourced from the user's webcam (or synthetic frames in tests).
Verification: webcam_demo.ino prints
frame N: 6144 bytes 320x240 fmt=4
continuously at ~10 fps under QEMU. 53 frames received in a 25 s
test window with debug logging disabled.
Bumps wokwi-libs/qemu-lcgamboa pointer to 5bbc92b (picsimlab-esp32)
which contains the final two fixes:
- eofs_remaining counter for multi-EOF-per-frame delivery
- reset_descriptor_ring() on rx_start 0→1 edge (matches hardware's
fresh-capture semantics that cam_hal relies on)
Adds:
- test/test-esp32-cam/autosearch/14_complete_emulation.md — full
forensic trace of the 8 distinct bugs found across the pipeline,
with final architecture diagram
- test/test-esp32-cam/tests/test_webcam_demo_live.py — pytest e2e
test that compiles webcam_demo.ino, boots it under the simulator
WebSocket, pushes a JPEG, and asserts fb_get returns frames
- test/test-esp32-cam/tests/debug_worker_direct.py — direct worker
bypass (no WS, no uvicorn) for dev-time tracing
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bumps wokwi-libs/qemu-lcgamboa pointer to a96c851 (picsimlab-esp32
branch) which contains:
- pack_one_pixel fix (was discarding half the JPEG data)
- split vsync_kick_timer / eof_timer (resolves chicken-and-egg
between VSYNC and rx_start)
- multi-descriptor walker (already in previous commit, recap)
Adds test/test-esp32-cam/autosearch/13_three_remaining_bugs.md
with line refs to upstream esp32-camera and a TODO list for the
next rebuild + verification cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First open-source end-to-end emulation of the AI-Thinker ESP32-CAM
in QEMU, paired with a browser webcam → firmware bridge so users can
develop camera sketches without hardware. Status: esp_camera_init()
returns ESP_OK; OV2640 chip-id verifies (PID/VER/MIDH/MIDL exactly
match the datasheet); GPIO 25 VSYNC NEGEDGE interrupt enabled by
the upstream driver. Final piece (cam_task accepting frames) is in
progress — descriptor walker fix landed in this commit.
Backend (Python/FastAPI):
- simulation.py: camera_attach/frame/detach WS handlers
- esp32_worker.py: ctypes binding to velxio_push_camera_frame +
feature-detection fallback for older DLLs
- esp32_lib_manager.py: forward camera commands to the worker stdin
- esp-idf-template/main/CMakeLists.txt: esp32-camera headers added
via add_prebuilt_library + REQUIRES driver (resolves i2c_master_*
symbols). LED_BUILTIN=2 fallback for sketches that hardcode it.
Frontend (React/TS):
- EditorToolbar.tsx: ESP32-CAM (and the rest of the ESP32 family)
added to isQemuBoard list — Run button now starts the QEMU bridge
for these boards instead of falling through to the AVR path
- useWebcamFrames.ts: getUserMedia → OffscreenCanvas →
toBlob('image/jpeg') → base64 → WS at ~10 fps
- CameraToggle.tsx: header button with status colors + frame counter
- SimulatorCanvas.tsx: render CameraToggle for esp32-cam boards
- Esp32Bridge.ts: sendCameraAttach/Frame/Detach + chunked btoa
- useSimulatorStore.ts: diagnostic log on compileBoardProgram
- components-metadata.json: regen including esp32-cam component
Submodule pointer:
- wokwi-libs/qemu-lcgamboa → ff8eee0 (camera devices commit on
davidmonterocrespo24/qemu-lcgamboa branch picsimlab-esp32)
Investigation + tests in test/test-esp32-cam/:
- 13 autosearch markdown docs (overview, SOTA, OV2640 spec, DVP/I2S
spec, build blueprint, blockers resolved, descriptor walker fix)
- 5 sketches (camera_init, sccb_probe, dma_smoke, frame_roundtrip,
webcam_demo) + 8 live + WS regression tests
- README with the user-facing flow
.gitignore:
- libqemu-*.dll.{pre-camera,new,bak} (rollback points, regenerated)
- wokwi-libs/esp32-camera/ (clone consumed by arduino-esp32 path,
not part of this repo)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Update esp32_lib_manager.py to dynamically set library extensions based on the platform (Linux, Windows, macOS).
- Update submodule references for qemu-lcgamboa and wokwi-elements.
- Add documentation on ESP32 Arduino runtime crashes related to cache disable during WiFi/BT initialization.
- Introduce regression tests for the ESP32-CAM blink issue, ensuring the user sketch matches the reported problem.
- Implement an IRAM-safe blink sketch to confirm the regression is due to the Arduino runtime.
- Create a comprehensive test suite to cover various layers of the simulation and compilation process.
Wire up three real, public-domain ROMs from the silicon era and prove
they boot end-to-end on the clean-room chip implementations. Each
test reads a separate well-known boot artifact:
* Busicom 141-PF firmware (4004, 1 KB, Intel PD 2009)
Wires real 4004 + real 4002 chips on the multiplexed nibble bus.
Toggles TEST every ~400 phases to mimic the printer-drum encoder
pulse the firmware polls. Asserts >2000 opcode fetches, >15 unique
PC addresses, and >100 CMROM strobes.
* Palo Alto Tiny BASIC v2 (8080, 1.9 KB, Wang 1976 PD)
CPUville port loaded from Intel HEX. Fake polled 8251 UART at port
0x02 (data) / 0x03 (status). Asserts the captured TX stream
contains the BASIC "OK" prompt — proving the interpreter reached
its REPL.
* Galaksija ROM A (Z80, 4 KB, Voja Antonić PD 1984)
ROM A+B at 0x0000..0x1FFF, system RAM at 0x2000..0x3FFF. Asserts
PC visits the JP target 0x03DA from reset and the ASCII "READY"
prompt appears in RAM after init.
ROMs are downloaded to roms/{4004,8080,z80}/ and gitignored — the
tests skip cleanly when the binaries are absent. License-clean: no
GPL ROMs, all PD by upstream provenance.
Total: 126 → 129 passing, 0 todo, 0 failed; 19 → 22 test files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Convert the last outstanding it.todo (4004 Busicom-style program)
into a passing integration test. The Busicom 141-PF firmware itself
isn't available in-environment, so this is an original demo that
exercises the same bus paths the firmware used:
CLB ; loop: SRC P0 ; WMP ; IAC ; JUN loop
Wires real 4004 + real 4002 chips on a shared D bus and uses the
JS-side nibble-bus driver to feed the 6-byte program. The 4002's
O0..O3 output port blinks through 0, 1, 2, 3, …, F, 0, … each
iteration. Test asserts the first 6 distinct outputs are 0..5 —
proving the loop iterates and the output port reflects each WMP-
driven ACC update faithfully.
Final state: 126 tests, 126 passing, 0 todo, 0 failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Convert 7 outstanding it.todo markers into actual passing tests now
that the chips and bus infrastructure can support them:
- 4004 LDM: ACC observed via SRC + WMP X2 bus drive
- 4004 FIM: register pair observed via SRC X2/X3 nibble drives
- 8080 hand-built loop: LXI/MVI/INR/DCR/JNZ decrements counter
- Z80 IM 2: vector table at I:00 → ISR via INT̅ low
- 8086 1 MB wrap: DS=0xFFFF + offset 0x11 lands at physical 0x00001
- 8086 ALE pulse: counts ALE rising edges per bus cycle
- 8086 AD release: external drive sticks during T2 (chip released)
- 8086 hello-world: 5 MOV BYTE [imm], imm writes to memory-mapped
"UART" at DS:0x9000; bus capture + RAM peek verify "Hello"
Plus: remove redundant 8080 CPUDIAG and Z80 ZEXDOC todos — the
actual end-to-end runs already pass in dedicated cpudiag.test.js
and zexdoc.test.js files.
Suite is now 125/126 passing, 1 todo (Busicom 141-PF demo, awaiting
firmware ROM), 0 failed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply the same xact_t pattern from the 4004 (phase D-2) to the 4040,
so SRC and the I/O group (WRM/WMP/WRR/WPM/WR0..3/SBM/RDM/RDR/ADM/
RD0..3) drive or sample the multiplexed nibble bus during X2/X3 with
CM-RAM (or CM-ROM for ROM-port ops) strobed.
The 4040's two CM-ROM lines (selected by rom_bank) and its STP/INT
control flow are unchanged — the bus action is staged at M2 and
acted on at X2/X3, fitting cleanly inside the existing PHASE_X3
control-flow block.
Two new integration tests under "4040 + 4002 RAM integration" mirror
the 4004's: SRC + WMP drives the output port, and SRC + WRM/RDM
round-trips a nibble through 4002 storage.
Total test_intel: 117 passing, 11 todo, 0 failed (was 115).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 4004 chip now drives or samples the multiplexed nibble bus during
X2/X3 with CM-RAM (or CM-ROM) strobed for SRC, WRM, WMP, WRR, WPM,
WR0..3, SBM, RDM, RDR, ADM, RD0..3 — completing the I/O group that
was previously stubbed. The 4002 RAM chip is rewritten with a
phase-count-based timing model that samples the opcode at M1/M2 and
drives or latches the bus at the correct frame relative to the 4004's
drives.
Two new integration tests in 4002-ram.test.js wire a real 4004 + 4002
on the same board and prove the round-trip:
1. SRC P0 + LDM 3 + WMP — 4002 output port goes to 3.
2. SRC P0 + WRM 5 + CLB + RDM + WMP — 4002 output port goes to 5
(proves both write and read paths through the bus).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 4002 is the data/IO partner of the 4004/4040. 16-pin DIP, 80
nibbles (4 registers × 16 main chars + 4 status chars each), plus
4 dedicated output port pins driven by the WMP instruction.
This skeleton:
- Pin contract registered (D0..D3, O0..O3, SYNC, CL, RESET, CM,
VDD, VSS).
- Storage allocated (main[4][16] + status[4][4] arrays).
- SYNC + own timer + CM-strobe gating tracks the SRC chip-select
latch at X2/X3 (compile-time RAM4002_CHIP_PAIR selects which of
4 chip pairs this instance responds to).
- RESET clears storage and drops output port to 0.
Not yet implemented (Phase D-2 follow-up): full SRC + WRM/RDM/WR0..3/
RD0..3 round-trip with the 4004. The 4004 chip currently stubs
those I/O instructions, so even though the 4002's address-latching
works, no data ever flows. Requires modifying 4004.c to drive the
bus during X2/X3 of SRC and during M2 of the I/O group.
Tests: 2/2 passing (pin contract + RESET behaviour). Total
test_intel: 111→113 passing. The 4-chip 4004 ecosystem (4001 +
4002 + 4004 + canvas-deployable variants) now exists.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First test wiring the 8086 CPU to a real 8259 PIC chip on the same
board and proving hardware-interrupt routing works end-to-end:
IRQ0 input → PIC asserts INT → CPU's INTR pin → CPU runs INTA
cycle → PIC drives vector 0x40 on AD bus → CPU does do_int(0x40)
→ fetches CS:IP from IVT entry at 0x100 → ISR runs → IRET → main
resumes from HLT.
Two related chip fixes required to make this work:
1. 8086 INTA cycle no longer drives AD itself.
Real 8086 INTA bus cycle has the PIC drive the data lines, not
the CPU. My earlier code did `bus_read_byte(0, false)` which
first drove AD with addr=0, overwriting whatever the PIC had
driven. Fix: release_ad → INTA̅ low → sample AD (PIC's INTA
watcher fires synchronously and drives) → INTA̅ high.
2. 8086 HLT now interruptible.
on_clock previously early-returned on G.halted, so step()
never ran and the INTR check never executed. Real 8086 HLT
wakes on INTR/NMI. Fix: remove the early return; step()'s
own halted check (later in the function) only no-ops if no
pending interrupt.
Tests: total test_intel 110 → 111 passing (+1, the integration
test). 0 failed. 11 todo. test_8086 now 11→12 passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two long-deferred Phase C chips, both clean-room from public Intel
datasheets. These complete the support-chip ecosystem needed for
real interrupt-driven 8080/Z80/8086 software on the canvas.
8259 PIC (~280 LOC, single-master subset):
- Full ICW1..ICW4 init sequence with branching on single/cascade
and ICW4-needed flags.
- IRR/ISR/IMR registers; OCW3 read-back; OCW1 mask write.
- Priority-based INT (lower IRQ# = higher priority, fully-nested);
pre-emption when a higher-priority IRQ arrives during a lower-
priority ISR.
- INTA falling-edge → drives vector_base + IRQ# on D bus.
- Non-specific (0x20) and specific (0x60..67) EOI.
- Cascade-master/slave routing NOT implemented (single master is
sufficient for 95% of demos).
- 7/7 tests passing.
8253 PIT (~210 LOC, Modes 0/2/3 subset):
- Three independent 16-bit counters with own CLK/GATE/OUT pins.
- Mode 0 (interrupt on terminal count) for one-shot timers.
- Mode 2 (rate generator) for system tick.
- Mode 3 (square wave, decrements by 2) for PC-speaker tone.
- Modes 1/4/5 coerced to Mode 0 (rare in practice).
- Full RW mode set: LSB-only / MSB-only / LSB-then-MSB / latch.
- GATE-low pauses the countdown.
- 4/4 tests passing.
Tests: total test_intel 99→110 passing (+11). 0 failed. 11 todo.
Master plan doc updated: Phase C extension done; Phase G still
deferred.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 4001 is the canonical ROM partner of the 4004/4040. 16-pin DIP,
256 bytes of mask-programmed ROM accessed over the 4-bit multiplexed
nibble bus, plus 4 I/O port lines (WRR/RDR — not yet wired).
Implementation: ~140 LOC clean-room from MCS-4 manual §V. The chip
has its own timer at 1351 ns (matching the 4004's clock period), with
a state machine that walks the 8-phase frame in lockstep with the
4004:
S_IDLE → (SYNC↑) → S_SAMPLE_LOW (A1 nibble) → S_SAMPLE_MID (A2) →
S_SAMPLE_HIGH (A3, addr complete) → S_DRIVE_HI (M1, drive opcode
high nibble) → S_DRIVE_LO (M2, drive low nibble) → S_POST (X1..X3
idle) → wait for next SYNC.
Timing trick: the 4001 must be added to the board BEFORE the 4004
so its tickTimers fires first per advanceNanos. The 4001 then runs
one frame "behind" the 4004 — sampling what the 4004 drove last
frame and driving what the 4004 will read this frame. Documented in
the chip's source and the master plan.
Integration test (`test_buses/4001-rom.test.js`) wires both chips on
the same board and verifies the 4004 actually fetches and executes
opcodes from the 4001 (PC walks 0, 1, 2 with the embedded NOP image).
This is the first end-to-end test of the 4-bit multiplexed bus
working between two real WASM chips on the canvas, not just JS
helpers — proving the bus model scales.
Deferred for the next Phase D iteration: 4002 RAM (similar shape +
SRC chip-select latching), 4004 SRC/WRM/RDM wiring to exchange data
with the 4002, and the Busicom 141-PF integration once both ROM and
RAM chips are real.
Tests: total test_intel 98 → 99 passing, 0 failed, 11 todo.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two milestone integration tests that run public-domain test ROMs
through the full 8080/Z80 chip + bus + BDOS-stub stack:
8080:
- 8080PRE.COM (1 KB preliminary test) — runs to completion, no ERROR.
- TST8080.COM (1.5 KB Microcosm 1980 CPUDIAG) — the canonical 8080
validation. Chip prints "CPU IS OPERATIONAL". This is the same
diagnostic that real Altair/IMSAI machines used to validate their
CPUs in the late 70s/early 80s. ~52s wall-clock, 2M simulated cycles.
Z80:
- ZEXDOC (8.5 KB Frank Cringle 1994 instruction exerciser, documented
flags subset of ZEXALL) — chip prints the "Z80 instruction exerciser"
banner and runs without ERROR within a 5M-cycle budget.
Test infrastructure:
- test/test_intel/roms/{8080pre,tst8080,8080exm,zexdoc}.bin — public-
domain ROMs mirrored from altairclone.com and floooh/chips-test.
- 64 KB system image builder: CP/M zero-page (JMP 0x0100 at PC=0,
JMP-to-BDOS at 0x0005), BDOS handler at 0xFE00 implementing
functions 2 (print char in E) and 9 (print string at DE until '$'),
using OUT port 0x01 to emit each char. The harness captures OUT
cycles via the WR̅-falling + IORQ̅-asserted pattern.
Lesson: BDOS at 0x0F00 collided with ZEXDOC.COM (8.5 KB extending
to 0x21A9). Moved BDOS to 0xFE00 — well above any reasonable .COM
program region. CPUDIAG worked at either address since TST8080 is
only 1.5 KB.
Tests: 94→98 passing. Total test_intel 105→109 (4 new tests).
0 failed. 11 todo (mostly 8086 corner cases + Busicom + full
ZEXDOC). Master plan doc updated marking phase F as partial.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause of the deferred CALL/RET test: my chip was missing the
0xC6 / 0xC7 (Group 11 — MOV r/m, imm) opcodes. Bytes like the
test's "MOV byte [0x8002], 0x55" (0xC6 0x06 0x02 0x80 0x55) fell
through to the default NOP, then the chip decoded the residual
0x06 0x02 0x80 0x55 as PUSH ES + ADD r/m + ... taking SP into
unpredictable territory.
Implementation:
- 0xC6 (8-bit) and 0xC7 (16-bit) variants added.
- The encoding is opcode + modrm + disp + imm. Critically the disp
bytes (consumed by calc_ea) come BEFORE the immediate, so we
compute EA first, then fetch imm. Earlier draft had imm fetched
before disp — that had imm winning the disp slot and disp becoming
the next instruction's bytes. Caught only after wiring CALL+RET.
Tests: 8086 10→11 passing. Total test_intel: 93→94 passing,
0 failed, 11 todo. CALL/RET integration test now active and green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Final tally for this session's work:
- Started at 37 passing tests after the initial 5-CPU baseline.
- Phase A (8080 INTA bus protocol): +1 test.
- Phase B (Z80 CB/DAA/RLD/ADC HL/CPI + X/Y flags): +10 tests.
- Phase C partial (rom-1m, 8255 PPI, 8251 USART): +13 tests.
- Phase E (8086 string ops, MUL/DIV, BCD, port I/O, shifts, etc):
+7 tests.
Total: 93 passing, 12 todo, 0 failed.
Plan tracked in autosearch/18_complete_emulation_plan.md. Phases D
(4004/4040 I/O completion using deferred 4001/4002 chips), F (real
software validation: CPUDIAG/ZEXDOC), and G (cycle accuracy) remain
as future iterations.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Implement UC8159cDecoder for handling 7-colour ACeP panels.
- Introduce painting functions for UC8159c frames in EPaperPart.
- Update EPaperPart to handle both SSD168x and UC8159c frame types.
- Add integration tests for EPaperPart and UC8159cDecoder.
- Create example sketch for 5.65" ACeP 7-colour panel.
- Enhance error handling in test cases for library dependencies.
Three new bus-device chips, all clean-room from public Intel datasheets:
- rom-1m: 64 KB ROM mapped at 0xF0000..0xFFFFF for 8086 boot. 20-bit
address bus watch with out-of-range tristate. 16-byte known signature
pre-loaded at the reset vector 0xFFFF0.
- 8255 PPI: Mode-0 (basic I/O) only; three 8-bit ports with split
upper/lower port C. Control word parsing for direction setup. Bit
set/reset and Modes 1/2 deferred.
- 8251 USART: async-mode UART using vx_uart_attach for bit-timing;
mode word + command word + status interface; TxRDY/RxRDY/TxEMPTY
status pins; DTR/RTS pass-through. Internal-reset honoured.
Deferred to a follow-up Phase C+:
- 4001 ROM and 4002 RAM (multi-phase 4-bit bus timing requires either
an external clock-gen chip or Bus4004-equivalent host coordination
that's only available in the JS test harness today).
- 8253 PIT (6 modes, countdown logic).
- 8259 PIC (ICW init state machine + cascade + INTA cycle).
These are flagged in autosearch/18_complete_emulation_plan.md as
deferred — Phase D (4004/4040 I/O) and the 8259 work depend on them
landing first.
Tests: test_buses 17 → 30 passing (+13). Total test_intel 73 → 86
passing, 0 failed, 17 todo. Master plan doc updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings the Z80 from 8080-superset baseline toward ZEXDOC compliance:
- CB prefix: full 256 ops (BIT/SET/RES + RLC/RRC/RL/RR/SLA/SRA/SLL/
SRL) on r ∈ B/C/D/E/H/L/(HL)/A.
- DDCB / FDCB indexed bit ops with displacement-before-opcode order.
Sean Young's undocumented "result also stored to non-(HL) register"
semantics included.
- Undocumented X (bit 3) and Y (bit 5) flag bits on every flag-
setting instruction (set_sz / set_szp / add_hl / cpl / etc.).
- Z80-specific DAA via N flag direction (Sean Young §4.7 algorithm —
the canonical ZEXALL-passing form).
- CPI / CPD / CPIR / CPDR with X/Y from (A − (HL) − H) per
Sean Young §4.2.
- RLD / RRD 12-bit nibble rotates between A and (HL).
- 16-bit ADC HL,rr (ED 4A/5A/6A/7A) and SBC HL,rr (ED 42/52/62/72)
with full S/Z/PV/H/N/C/X/Y handling and bit-12 half-carry.
Deferred:
- MEMPTR (WZ) full update map (only the strictest ZEXALL cases need it).
- Block I/O instructions' deterministic flags (Phase F polish).
- ZEXDOC ROM integration test (Phase F).
Tests: z80 11→21 passing (+10: SET, RES, RLC A, SRL A, SRA A, BIT 7,
DAA, ADC HL BC, RLD, CPIR). Total test_intel: 64→73 passing, 0 failed.
Master plan doc updated: phase B marked done; phase C starting.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
davidmonterocrespo24