Closes the register-then-immediately-forgot race for good. Two parallel
calls to /velxio/api/send-welcome and /velxio/api/send-password-reset
were hitting Odoo's REPEATABLE-READ snapshot timing: both workers took
their snapshot BEFORE either had committed the partner row, so each
ran its own INSERT and the second blew up on the velxio_user_id
unique constraint — costing one of the two emails.
Add a new sync_partner() helper that POSTs to a new Odoo route
/velxio/api/upsert-partner (lives in velxio_subscription) which only
upserts the partner — no mail, no subscription, fast. Register and
forgot-password await this BEFORE firing the async welcome / reset
tasks. Each user's flow is sequential at the Velxio HTTP layer, so
the snapshot race disappears.
sync_partner() reuses the existing _post() error-swallowing pattern.
If Odoo is down, sync_partner returns None and the await is a no-op
— the user still registers / gets the generic 200, the welcome /
reset endpoints retain their defensive upsert as a fallback path.
Adds velxio_user_id to the send_password_reset payload (mirroring
send_welcome). The Odoo side's res_partner.velxio_user_id is unique,
so when Odoo eventually upserts on this endpoint the constraint
serializes concurrent register-then-immediately-forgot upserts and
prevents the duplicate-partner record the previous wire format risked.
Closes the remaining gaps in cross-board I2C so any topology of
supported boards (Uno↔ESP32, two ESP32s, Uno↔Uno↔Uno, ESP32-C3
connected to anything, etc.) works end-to-end with all I2C
components including write-only sinks (SSD1306, PCF8574, LCD-I2C).
Implementation (6 phases):
1. **BFS routing in I2CBusManager**: connectToSlave + handleExternalConnect
walk the bridge graph with a visited Set so multi-hop chains
(A↔B↔C with the device on C) resolve transparently. A new
forwarder-device shim is installed at intermediate hops so the
existing handleExternalWrite/Read/Stop machinery routes
through without per-method visited tracking.
2. **Per-peer proxy ownership in Esp32BridgeShim**: replaces the
global _proxiedAddrs Set with _proxiedByPeer Map so concurrent
bridges to the same ESP32 (e.g. wired to both Uno and Pico)
don't wipe each other's proxies on teardown. Interconnect's
per-wire teardown calls clearProxiesForPeer(peerBus) instead of
clearAllProxies.
3. **BFS-aware proxy sync**: syncProxyFromPeer now walks the peer
bus + its transitive bridges, so an ESP32 sees devices on
boards two or more hops away. _peerDeviceLookup keeps a flat
addr → device map for write-forwarding and resync.
4. **Periodic resync (250 ms)**: Esp32BridgeShim runs a setInterval
while any proxy is live, re-dumping each device with
dumpRegisters() and pushing updateProxyI2c only when an XOR-
stride hash changes. This keeps RTC time advancing visible to
ESP32 firmware without flooding the WS pipe with static
calibration dumps. Hash is primed during initial sync so the
first tick doesn't push a redundant identical buffer.
5. **Write-forwarding ProxySlave → peer**: backend ProxySlave
buffers write bytes during the transaction and emits a
`proxy_i2c_complete` event on STOP / repeated-START. Frontend
Esp32Bridge dispatches the event to a new onProxyI2cComplete
callback; the shim replays the byte sequence on the actual
peer I2CDevice via writeByte() + stop(). Makes ESP32 firmware
writes to peer SSD1306 actually repaint the OLED, peer PCF8574
latch updates, peer I2CMemoryDevice register mutations propagate.
6. **ESP32-C3 routed as bridge**: Interconnect.isBrowserSim no
longer claims c3/xiao-c3/c3-supermini — they were already
going through Esp32Bridge per the store's ESP32_RISCV_KINDS
routing, but Interconnect was treating them as browser sims
which broke proxy install. isEsp32Bridge now correctly
includes c3 family + ESP32-S3 + Arduino Nano ESP32.
Defensive: addBoard now disposes any existing shim's proxies
before overwriting simulatorMap entry so test reruns don't leak
timers.
Tests:
- 4 BFS multi-hop tests (i2c-multi-board-slave-gap.test.ts)
- 11 cross-board scenarios + per-peer + write-forward + resync
(i2c-esp32-multiboard-bridge.test.ts)
- 1 real-firmware E2E for write-forward via QEMU (compile +
load + observe proxy_i2c_complete arriving with the byte)
- New sketch fixture: esp32_i2c_write_to_peer.ino
Result: 90 test files / 1295 tests pass / 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the transactional email pipeline driven from the Odoo SMTP relay so
new sign-ups get a Velxio-branded welcome and existing users can reset a
forgotten password without us running our own outbound mail server.
Backend:
- PasswordResetToken model: one-time, SHA-256-hashed (plain text never on
disk), TTL 60 min, marked used_at on consume to prevent replay.
- POST /auth/forgot-password — anti-enumeration (always 200 + generic
message), rate-limited 3/hour/user.
- POST /auth/reset-password — verifies token, hashes new password,
atomically marks token used.
- /auth/register hooked with asyncio.create_task to fire welcome mail —
registration is never blocked on Odoo being up.
- New service app/services/odoo_mail.py: async httpx wrapper, fire-and-
forget, swallows every error so the request lifecycle stays clean.
- Settings ODOO_URL / ODOO_API_KEY / ODOO_MAIL_TIMEOUT_S /
PASSWORD_RESET_TOKEN_TTL_MINUTES / PASSWORD_RESET_RATE_LIMIT_PER_HOUR.
Frontend:
- /forgot-password page (single email field + "check your inbox" state).
- /reset-password?token=XYZ page (new password + confirmation, redirects
to /login?reset=ok on success).
- "Forgot your password?" link + green confirmation banner on /login.
- authService gains requestPasswordReset() and resetPassword().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Implemented `i2c-esp32-real-firmware.test.ts` to test ESP32 I2C communication via backend and WebSocket.
- Created `load-example-transitions.test.ts` to ensure proper loading of examples between board-less and board-based contexts.
- Added `CircuitVerificationModal.tsx` to display circuit verification results before running simulations.
- Developed `circuitVerifier.ts` to perform pre-flight checks for circuit safety, identifying potential issues like short circuits and component overloads.
- Introduced minimal ESP32 I2C master sketch `esp32_i2c_writer.ino` for testing I2C transactions.
A user reported on Discord: "the Velxio Console doesn't update anything,
it just waits until the very end and displays everything in one go".
True for the async compile path — /compile/status only carried `state`
and the final `result`, so the editor's CompilationConsole stayed empty
during the 5-7 minute cold ESP-IDF builds and dumped 1500 lines at once
when the build finished.
This wires live build output through the whole stack.
Backend (espidf_compiler.py)
- New _run_with_streaming() helper. When a progress_callback is provided
it spawns the subprocess via Popen + stdout/stderr drain threads and
invokes the callback line-by-line. When None it falls back to the
existing subprocess.run(capture_output=True) one-shot path so the
unit-test code that doesn't care about live output is unaffected.
- compile() and _compile_in_dir() take an optional ProgressCallback.
- _run_cmake / _run_ninja closures now go through _run_with_streaming
with that callback. cmake configure (~2-5 s) + ninja (~5-300+ s) both
stream now; the ninja output is the one users actually want to watch.
Backend (compile.py)
- _compile_job seeds COMPILE_JOBS[id]['stdout_buffer'] = '' and defines
on_progress_line(line) which appends to it. Buffer capped at 256 KB
(tail kept) so a runaway build can't OOM the FastAPI process.
- The buffer is preserved on both the success and the error path so
late polls still see the log even after state transitions to
done/error.
- /compile/status now returns the buffer as a `stdout` field.
CompileStatusResponse gains the field with default '' so old clients
that don't read it still work.
Frontend (compilation.ts)
- compileCode() takes a 4th argument: optional CompileProgress
callback fired every poll while state ∈ {pending, running}. Carries
the cumulative stdout (caller computes deltas) plus elapsed seconds.
- Surfaces the new `stdout` field of /compile/status and forwards it
to the callback. Errors thrown from the callback are swallowed —
a faulty UI hook must never break the polling loop.
Frontend (EditorToolbar.tsx)
- Both compileCode() call sites (Run and Compile-All) now pass an
onProgress callback. It tracks `lastStreamedLen` per-compile, splits
each new delta on newlines, and appends them as `info`-typed
CompilationLog entries via setCompileLogs. The Compile-All flow
prefixes each line with the board label so multi-board builds stay
readable.
- After the build settles, the existing parseCompileResult call still
runs and appends the structured analysis on top of the live stream
— that's where FAILED-block detection + the `error`-typed entries
that drive the auto-switch-to-errors filter live.
Net effect on the user complaint: cold ESP-IDF builds now show the
ninja [N/1483] progress lines streaming into the console as they
happen, instead of staring at an empty panel for 5-7 minutes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three coordinated fixes that together close the "ESP-IDF compile takes
5-7 min every time" gap and prevent the failure mode where a user clicking
compile multiple times spawns six ninja processes that peel each other
apart on a modest VPS.
What was wrong
- /compile/start generated a fresh uuid4 every call, so 6 clicks = 6
independent builds racing each other. Saw load average 30 on the prod
VPS during a real BMP280 attempt today.
- No concurrency limit anywhere; asyncio.create_task() fired without
gating.
- ccache was wired in last week (PR #149) but reported 18,350 cacheable
calls and **0 hits** because the build dir was a fresh
tempfile.TemporaryDirectory(prefix='espidf_') per compile. The random
/tmp/espidf_<random>/ path baked into -I and -fmacro-prefix-map flags
→ different command line every compile → ccache hash miss every time.
What this PR does
1. Job deduplication (`backend/app/api/routes/compile.py`)
- New `_job_key(files, board_fqbn)` returns SHA-256 of normalised file
names + contents + board. Order-independent.
- New `JOB_BY_KEY: dict[str, str]` indexes hash → job_id.
- `compile_start` checks JOB_BY_KEY before spawning a new task; if a
job for this exact content is already pending or running, returns
the existing job_id (logs `[compile] dedup hit — reusing job <id>`).
- `_purge_expired_jobs` evicts both COMPILE_JOBS and JOB_BY_KEY,
keeping the index consistent. Edge case where two jobs share a key
(old finished, new running) is handled — only evict the key entry
if it still points at the purged job.
2. Concurrency control (`backend/app/api/routes/compile.py`)
- `_COMPILE_SEMAPHORE = asyncio.Semaphore(2)` global cap on
simultaneous compiles.
- `_target_lock(board_fqbn)` returns a per-target asyncio.Lock so
concurrent compiles to the SAME board (sharing the persistent build
dir) serialise. Different boards still run in parallel up to the
semaphore cap.
- `_compile_job` acquires sema → per-target lock → flips state to
`running` → calls `_run_compile`. Pending state now accurately
reflects "queued waiting for resources".
3. Persistent build dir (`backend/app/services/espidf_compiler.py`)
- New `_prepare_persistent_project_dir(idf_target)` materialises
`/var/lib/velxio-build/<target>/project/` from the template on
first use; on subsequent compiles it wipes only `main/` and
`user_libs/` (the per-compile parts) and leaves `build/` alone so
ninja's incremental cache + ccache .o files survive.
- Toolchain version sentinel (`.idf_version`) wipes the whole target
dir if the ESP-IDF or arduino-esp32 version changes — cached
objects from the old toolchain are no longer ABI-compatible.
- `compile()` is now a thin dispatcher: persistent path or fallback
to the legacy `tempfile.TemporaryDirectory()` flow. The actual
build logic was extracted into `_compile_in_dir()` so both paths
share one implementation, no duplication.
- Escape hatch: `VELXIO_PERSISTENT_BUILD_DIR=0` env var falls back
to the tempfile path without rebuilding the image. Critical for
production safety.
4. ccache normalisation (`Dockerfile.standalone`)
- + `ENV CCACHE_BASEDIR=/var/lib/velxio-build` makes ccache canonicalise
absolute paths under that prefix when computing the cache key.
Robustens hits against any future subdir rearrangement.
5. Docker compose (`docker-compose.yml`)
- + named volume `velxio-build:/var/lib/velxio-build` so the persistent
build dir survives `docker compose up -d --build`.
- + env `VELXIO_PERSISTENT_BUILD_DIR=1` (default ON; users disable
without rebuilding).
Expected impact
- Cold first compile per container per target: unchanged (~5-7 min).
- Same sketch re-compiled: ~2-5 s (everything cached).
- Different sketch, same target: ~5-30 s (only user code + new lib steps
rebuild; ESP-IDF base hits cache).
- Different sketch with new libraries: ~30-90 s (new lib component
compiles; rest hits cache).
- Concurrent clicks on same example: 1 build, others poll the same
job_id. No more six-ninja meltdown.
Tests
- `test/backend/unit/test_compile_dedup.py` covers `_job_key` stability +
variance and `_purge_expired_jobs` consistency (including the
"two jobs share a key" edge case).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cold first compile per container is unchanged (cache empty). Subsequent
compiles drop from ~5-7 minutes to ~30-60 seconds because every ESP-IDF
base object (FreeRTOS, lwIP, esp_wifi, libsodium, soc, hal, …) hits the
cache. The user's BMP280 example, which hangs on cold compile, completes
near-instantly on the second attempt.
Why a transparent cache is safe: ccache hashes the preprocessed source +
flags + compiler. A cache hit only happens when the input is byte-for-byte
identical to a prior compile. Different sketches with different libraries
still get correct cache misses; there is no path where one project's
output contaminates another.
Changes
- Dockerfile.standalone: install ccache, set CCACHE_DIR=/var/cache/ccache,
IDF_CCACHE_ENABLE=1, configure 2 GB cap with compression. Compression
(level 6) cuts cache disk usage by ~40% with negligible CPU overhead.
- docker-compose.yml: named volume `ccache:/var/cache/ccache` so the
cache survives `docker compose up -d --build` (without it, every image
rebuild discards the cache).
- backend/app/services/espidf_compiler.py: pass `-DCCACHE_ENABLE=1` to
cmake when IDF_CCACHE_ENABLE is truthy. ESP-IDF's project.cmake
(`set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)` on line 374)
is what actually wires ccache in; without the cmake -D flag the env
var alone has no effect because we don't go through idf.py.
Escape hatch: set IDF_CCACHE_ENABLE=0 in compose env to disable without
rebuilding the image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The synchronous /api/compile endpoint forced one long-lived HTTP request
to span the entire build. Cloudflare's 100s edge timeout cuts that off
mid-flight for any cold ESP-IDF compile (BMP280 takes 5-7 min on first
run). The user-visible symptom was HTTP 524 well before the backend
even noticed.
Backend (compile.py)
- New `POST /api/compile/start` returns `{job_id}` immediately and
spawns the actual compile as an asyncio.create_task background.
- New `GET /api/compile/status/{job_id}` returns the current job state
(`pending` | `running` | `done` | `error`). Each poll completes in
milliseconds, far under any edge timeout.
- Existing `POST /api/compile/` kept verbatim for backward compatibility
(AVR/RP2040 builds finish in seconds and don't trip 524).
- Build logic extracted into `_run_compile()` so both paths share one
implementation; no duplicated ESP-IDF / arduino-cli branching.
- Async path opens its own short-lived DB session via AsyncSessionLocal
for metric recording — the request-scoped session is dead by the time
the background task finishes.
- COMPILE_JOBS dict purges entries 30 minutes after completion so a
busy server doesn't grow unboundedly.
Frontend (compilation.ts)
- compileCode() now: POST /compile/start → poll /compile/status every 2s
until state ∈ {done, error}, with a 15-minute client-side cap.
- 30s axios timeout per individual call (not per build) so transient
network blips during a long compile auto-retry instead of failing.
- 404 on /status throws (job expired / server restarted); other poll
errors warn and retry. Surfaces structured error responses verbatim
so the editor's compile-error panel keeps working unchanged.
Limitation: COMPILE_JOBS lives in-process; if velxio ever scales to
multiple FastAPI workers this needs to move to Redis or sqlite. Single-
instance is fine today.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The ESP32 BMP280 example compile was timing out at 98% (1473/1483 build
steps), failing with the unhelpful "ESP-IDF build timed out (300s)"
message even though every individual step was healthy.
Cold ESP-IDF builds that pull in external Arduino libraries — Adafruit
BMP280 + Adafruit BusIO + Adafruit Unified Sensor on top of the base
arduino-esp32 component tree — routinely produce ~1480 build objects.
On modest VPS hardware this takes 5-7 minutes the first time. Ninja's
incremental cache makes subsequent compiles seconds, but the first one
needs more headroom.
Constant lifted to NINJA_TIMEOUT_S so the value used in the timeout
matches the value reported in the error message — the previous code
hard-coded "300s" in two places that were free to drift apart.
Repro before: open the example "ESP32 — BMP280 Barometric Pressure"
on velxio.dev/editor on a clean container, click compile → fails after
5 minutes with timeout. After: completes in ~6 minutes on the first
run, ~5 seconds on subsequent runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI's e2e test_hcsr04_simulation.mjs caught the regression introduced by
a3f21a2 (the issue #101 fix). The worker crashes on boot with
Firmware decode error: No module named 'app'
esp32_worker.py runs as a subprocess via subprocess.Popen([sys.executable,
WORKER_PATH, ...]). When Python launches a script directly, sys.path[0]
is the SCRIPT's directory (backend/app/services/), not the backend root.
So `from app.services.esp32_flash_image import pad_to_flash_size` fails
because there is no `app/` under `backend/app/services/`.
esp32_lib_bridge.py wasn't affected because it runs in-process inside
uvicorn, where backend/ is implicitly on sys.path.
Fix: same try/except + importlib fallback the worker already uses for
esp32_i2c_slaves at the top of the file. First try the package import
(works when imported by the bridge's tests or anything else with the
backend root on sys.path), fall back to direct file loading otherwise.
Verified the fallback works in isolation by simulating the subprocess
context (sys.path containing only backend/app/services/) — the package
import fails as expected and the file-load fallback returns a properly
padded 4 MB buffer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A user reported the LEDC PWM RGB example failing to compile. The sketch
calls ledcAttach(pin, freq, resolution) — the one-shot API added in
arduino-esp32 3.x. Our toolchain image pins arduino-esp32 to 2.0.17
(matched to ESP-IDF 4.4.7 + the lcgamboa QEMU ROM), where the API is
the older two-step ledcSetup + ledcAttachPin pair. Sketches written
against 3.x docs hit "ledcAttach was not declared in this scope".
Bumping arduino-esp32 to 3.x means moving to ESP-IDF 5.x, which may
break our QEMU fork. Cheaper fix: ship a compat shim header in the
ESP-IDF project template that defines ledcAttach + ledcAttachChannel
in terms of the 2.x API, gated on `!defined(ledcAttach)` so it
disappears the day we bump.
espidf_compiler.py now injects #include "velxio_compat.h" right after
Arduino.h whether the user explicitly included Arduino.h or we
prepended it ourselves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two reports rolled into one sdkconfig change:
1. Beta testers reported "weird serial output" across DHT22, Servo+Pot,
Joystick, WiFi×2, and DualADC examples. The user's Serial.print lines
came interleaved with internal ESP-IDF chatter:
I (53306) gpio: GPIO[4]| InputEn: 1| OutputEn: 0| ...
I (10626) phy_init: phy_version 4791,2c4672b,...
sdkconfig.defaults didn't pin a default log level so it inherited
CONFIG_LOG_DEFAULT_LEVEL_INFO. Set WARN (level 2) so only warnings
and errors leak into the user's serial output. Sketches can still
esp_log_level_set() per tag at runtime if they want verbose.
2. BLE Advertise example failed to compile. sdkconfig had NimBLE
enabled, but arduino-esp32 2.0.17's BLEDevice.h targets Bluedroid.
Switch the stack: enable BT_BLUEDROID + BTDM_CTRL_MODE_BR_EDR_BLE +
BT_BLE so the standard arduino-esp32 BLE library compiles. Sketches
that explicitly include NimBLEDevice.h will not compile under this
config — that's a smaller minority than the BLEDevice.h users.
The runtime side of BLE still depends on stubs in the qemu-lcgamboa
fork; this only fixes the compile path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Issue #101 reproducer: an ESP32 sketch that pulls in Adafruit_SSD1306 +
Adafruit_GFX produced a "No response from server. Is the backend
running on port 8001?" error in the browser. The compile actually
succeeded backend-side, but the JSON response carrying the firmware
was ~5.5 MB of base64 — the ESP-IDF compiler builds a full 4 MB merged
flash image (mostly 0xFF padding), encodes it whole, and ships it. In
prod that response goes through nginx + Cloudflare, which buffer-fail
or RST the connection on payloads that big — axios then lands in the
"no response" branch with no HTTP status to surface.
Fix: trim the trailing 0xFF padding before serializing, re-pad to a
valid QEMU flash size (2/4/8/16 MB) just before mtd attach. Lossless:
bytes after `last_used` in the merge are 0xFF by construction, so
trim → pad reproduces the original image byte-for-byte.
Numbers from the reproducer (Adafruit_SSD1306 + Adafruit_GFX,
esp32:esp32:esp32 board):
before: ~5.5 MB JSON response
after: 539 KB JSON response (10× smaller)
backend/app/services/espidf_compiler.py
_merge_flash_image now tracks `last_used` across the three placed
sections (bootloader / partitions / app) and writes only
flash[:last_used] to merged_flash.bin.
backend/app/services/esp32_flash_image.py (new)
Shared `pad_to_flash_size(bytes) -> bytes` helper. Rounds up to the
next valid QEMU flash size with a 4 MB minimum, matches the
frontend's existing padToFlashSize logic in Esp32MicroPythonLoader.
Raises ValueError on >16 MB inputs (would indicate a broken upstream
merge, not anything user-recoverable).
backend/app/services/esp32_lib_bridge.py
backend/app/services/esp32_worker.py
Both QEMU consumer paths (in-process and subprocess) call
pad_to_flash_size right after base64.b64decode, before writing the
tmp .bin that QEMU attaches with `-drive if=mtd,format=raw`.
Verified: smoke test confirms trim → pad → original is byte-exact.
Edge cases covered: small payloads pad up to the 4 MB minimum;
firmwares >16 MB are rejected loudly.
Closes#101
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 4 nullable columns to the users table so deployments that wire an
external billing system (e.g. velxio.dev with Odoo) have a place to cache
subscription status. Self-hosters never write these — defaults are safe
(no paid features unlock).
- models/user.py: is_paid_subscriber (bool, default false), subscription_status
(str|None), subscription_period_end (datetime|None), odoo_partner_id
(int|None, indexed).
- main.py: 4 ALTER TABLE statements appended to the legacy_migrations list
so existing deployments auto-migrate on next boot.
- schemas/auth.py: extend UserResponse with the 3 user-visible fields
(is_paid_subscriber, subscription_status, subscription_period_end). The
frontend useAuthStore already persists the whole UserResponse, so these
surface automatically without any frontend changes upstream.
odoo_partner_id stays internal — clients don't need it.
Zero behavioural change for existing OSS deployments.
Three small, backwards-compatible hooks let anyone with private features
(velxio.dev's analytics, custom integrations, paid tiers, …) layer them
on top of the open-source build without forking files.
Backend (app/main.py):
- After standard router registration, try-import an optional `app.pro`
module exposing `register_pro(app)`. ImportError is silently swallowed
(the OSS image doesn't ship `app.pro`, so this is a no-op there).
Frontend:
- EditorToolbar: new optional `rightSlot` prop renders extra elements
after the built-in right-group buttons (mirrors the existing
`centerSlot` pattern).
- main.tsx: dynamic `import('@pro/index')` gated by VITE_PRO_BUILD env.
When unset (OSS build), the branch is dead-code-eliminated and no pro
chunk is emitted.
- vite.config.ts: `@pro` alias resolves to `src/__pro_stub__/` by default.
Private builds set `VITE_PRO_BUILD=true` and `PRO_OVERLAY_PATH=<path>`
to point at their real overlay tree.
- src/__pro_stub__/index.ts: 1-line no-op `mountPro` so TypeScript and
Vite resolvers stay happy in OSS builds.
Verified: `npm run build:docker` succeeds; `npm test` passes 1161/1162;
the OSS bundle (43 MB) contains zero references to `__pro_stub__`,
`@pro`, or `pro/index` (verified via `grep dist/`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The _should_include filter inside _merge_arduino_libs_to_component
rejected every subdirectory of a library except 'utility/' when the
library lacked a src/ layout. That blocked legitimate header dirs like
Adafruit_GFX_Library/Fonts/, breaking compiles that use any GxEPD2
example with a custom font (#include <Fonts/FreeMonoBold12pt7b.h>).
The earlier filter at the top of the function already excludes
docs/examples/tests/etc. via excluded_dirs, so anything that survives
that check is presumed to be buildable source. Letting all remaining
subdirs through restores Fonts/, gfxfont/, and similar conventional
auxiliary header directories that Adafruit-style libs rely on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The transitive-include scan in _merge_arduino_libs_to_component used
glob('*.h') on the component dir, which only finds headers at the root.
Libraries with a src/ layout (GxEPD2, ArduinoJson, most modern Arduino
libs) keep their headers under src/<...>, so the scan saw zero headers
and never queued their transitive deps.
Symptom: compiling a sketch that includes GxEPD2_3C.h failed with
'Adafruit_GFX.h: No such file or directory' even though Adafruit_GFX
was installed via the Library Manager — because the BFS never reached
its header from inside GxEPD2_GFX.h.
Switching to rglob('*.h') walks the full directory tree and lets the
BFS pick up Adafruit_GFX, Adafruit_BusIO, and any other transitive
dependency that lives under src/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves several install pain points reported by users (#108, #120) and
removes the obligatory upstream-clone step that confused contributors and
slowed down every Docker build.
Install fixes:
- nginx: server_name → catch-all default_server, drop Debian's stock site
so reverse-proxied users no longer get the "Welcome to nginx" page.
- entrypoint: auto-generate SECRET_KEY at first boot, persisted under
data/.secret_key. backend/.env is now optional in docker-compose.yml.
- backend: add greenlet>=3.0.0 (SQLAlchemy async dep that was missing on
some Python builds — caused uvicorn startup failures on WSL).
Wokwi libs come from npm:
- @wokwi/elements 1.9.2, avr8js 0.21.0, rp2040js 1.3.2 are pinned in
frontend/package.json. Vite aliases removed.
- Dockerfile.standalone no longer clones avr8js / rp2040js / wokwi-elements
/ wokwi-boards. Frontend stage is just COPY + npm install + build:docker.
- Board SVGs vendored under frontend/public/boards/ (10 deduped against
existing files, 2 truly new). third-party/wokwi-* clones become reference-
only credits — generate-component-metadata.ts skips gracefully when absent.
Production config split out:
- docker-compose.prod.yml, deploy/nginx.prod.conf, nginx-host-velxio*.conf,
update-third-party.bat removed. Production deployment lives in its own
repo: https://github.com/velxio/velxio-prod (host nginx + HTTPS + backups
+ pinned upstream commit).
Verified locally: 1161 frontend tests pass, build:docker completes clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The directory grew well beyond Wokwi-only contents: it now hosts
lcgamboa's QEMU fork (qemu-lcgamboa), Espressif's esp32-camera, the
ngspice WASM build, fritzing-parts, picowi, an alternative QEMU
(qemu-esp32), the 100_Days_100_IoT_Projects examples repo, and
Wokwi's own avr8js/rp2040js/wokwi-elements/wokwi-features/wokwi-boards.
"wokwi-libs" was misleading — half the contents have nothing to do
with Wokwi. "third-party/" is the standard convention for vendored
external dependencies.
Mechanical changes:
Path rename:
wokwi-libs/ → third-party/
update-wokwi-libs.bat → update-third-party.bat
docs/WOKWI_LIBS.md → docs/THIRD_PARTY.md
Submodule reconfiguration:
.gitmodules — 4 path= and section names updated
.git/modules/wokwi-libs/ → .git/modules/third-party/
each submodule's .git file rewired to ../../.git/modules/third-party/<name>
Reference updates (~80 files): vite.config.ts aliases, Dockerfile
COPY paths, GH Actions workflow steps, build_qemu_*.sh, all
docs/* and test/*/autosearch/* entries that mention the path,
package-lock.json file: dependencies, .gitignore patterns,
sitemap.xml + index.html SEO blurbs, scripts/generate-component-*,
.dockerignore, .idea/vcs.xml. Bulk replaced both `wokwi-libs/`
(path) and bare `wokwi-libs` (textual mentions in docs/comments).
Verified:
- npx tsc -b --noEmit produces no new errors related to these paths
- vite.config.ts aliases now point at ../third-party/avr8js etc.
- All 4 git submodules (avr8js, rp2040js, wokwi-elements,
wokwi-features) are linked under third-party/ with their
worktrees re-populated and config files referencing the new path
- `grep -r wokwi-libs` returns zero hits outside node_modules,
.vite, frontend/dist, third-party/ (upstream submodule contents),
*.pyc caches, and *.dll.pre-camera rollback binaries
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User reported only 2 frames rendering after the SPI batching change.
Diagnosis: the previous flush triggers (CS-line HIGH, buffer >=4096)
were both event-driven. Adafruit_ILI9341's ESP32 backend manages CS
via digitalWrite — i.e. through the GPIO peripheral, NOT the SPI
peripheral's hardware CS pin. So the CS-line-HIGH event from
picsimlab_spi_event NEVER fires for this driver. The buffer only
flushes when it hits 4096 bytes.
Frame 1: 38 400 bytes from drawRGBBitmap → 9 flushes at 4096-byte
boundaries → last 192 bytes stay in the buffer. Status bar adds
some bytes too → maybe one more flush.
Frame 2: same. But by frame 3 the firmware is running ahead of the
flush rhythm and somehow the buffer pattern wedges in a state where
no flush completes (likely a partial buffer that sits between
transactions while the firmware briefly waits on the next fb_get).
Hard to reproduce deterministically — but the symptom matches.
Fix: add a 50 ms periodic flush thread. Independent of any event,
it acquires the lock and flushes whatever's pending. Bounds the
worst-case latency at 50 ms (= 20 fps ceiling, more than enough for
the emulator).
Triple-trigger now:
1. CS HIGH (fast path for hardware-CS drivers)
2. Buffer >= 4096 (safety for big transactions)
3. 50 ms timer (catches GPIO-CS drivers, prevents stalls)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User reported the ESP32-CAM + ILI9341 live preview at ~1 frame/min.
Profile: 80×60 preview pushes 9600 SPI bytes per drawRGBBitmap, and
each byte was emitting a full {type:'spi_event'} JSON message over
the worker→backend→WS→frontend pipeline. Per-byte overhead ~150-200µs
in Python (json.dumps + sys.stdout.write+flush dominates) plus
asyncio + WS dispatch. Net: 1.5-2 sec/frame minimum, much worse with
GIL contention.
Fix: buffer MOSI bytes in the worker and emit a single base64-encoded
`spi_batch` message when CS goes HIGH (transaction ended) or the
buffer crosses 4 KiB. ~9600 events/frame collapse to ~3 messages.
backend/app/services/esp32_worker.py:_on_spi_event
- Add _spi_byte_buf bytearray + threading.Lock
- On op==0x00 (byte): append; flush early if buf >= 4096
- On op==0x01 (CS change): flush buffer, then emit the CS event
via the legacy spi_event channel (ePaper / custom chips that
observe CS still get it).
frontend/src/simulation/Esp32Bridge.ts
- New 'spi_batch' message handler decodes b64 and replays each
byte through the existing onSpiByte callback. Parts that
subscribed via simulator.spi.onByte don't notice the protocol
change. The 'spi_event' branch still handles CS changes plus
legacy single-byte payloads for backwards compat.
Now that 38 KB/frame is cheap, restore preview to 160×120 + JPEG
quality 0.35 in the gallery example. Real measured speedup: ~50× on
the QVGA preview demo. Real hardware was never affected — it runs
SPI at 80 MHz and pushes the bitmap in ~4 ms either way.
PSRAM emulation is unrelated to this bottleneck and was left untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User confirmed the FULL pipeline works with their laptop webcam at
QVGA quality 0.6 — frontend → WS → backend → worker → DLL → I²S →
firmware → esp_camera_fb_get() → user sketch.
[Frame #1] 8192 bytes 320x240 fmt=4
├─ SOI (FF D8 FF): ✓ at offset 0
├─ EOI (FF D9): ✓ at offset 8190
└─ First 16 bytes: FF D8 FF E0 00 10 4A 46 49 46 …
(JFIF, real webcam)
Stats: 10/10 Valid JPEGs at ~3.5 fps.
Changes:
- wokwi-libs/qemu-lcgamboa @ e4321d1 (picsimlab-esp32):
EOFS_PER_FRAME 6→8, inject_eoi_now flag for EOI injection on the
last EOF of each VSYNC burst. Handles JPEGs of arbitrary size by
forcing FF D9 at offset 8190 — JPEG decoders tolerate the
truncation gracefully.
- backend/esp32_worker.py: throttled trace log every 30 frames
(`camera_frame #N received (NNNN bytes)`) so users can confirm
the frontend → worker leg is alive without flooding the log.
- test/test-esp32-cam/autosearch/14: documented bug #9 (the 9th and
final silent bug — real webcam JPEGs exceed the deliverable byte
budget) with full forensic trace + final architecture diagram.
The emulation now handles ANY user webcam → ESP32-CAM use case end
to end. Standard upstream esp_camera_init() / esp_camera_fb_get()
sketches work without modification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First open-source end-to-end emulation of the AI-Thinker ESP32-CAM
in QEMU, paired with a browser webcam → firmware bridge so users can
develop camera sketches without hardware. Status: esp_camera_init()
returns ESP_OK; OV2640 chip-id verifies (PID/VER/MIDH/MIDL exactly
match the datasheet); GPIO 25 VSYNC NEGEDGE interrupt enabled by
the upstream driver. Final piece (cam_task accepting frames) is in
progress — descriptor walker fix landed in this commit.
Backend (Python/FastAPI):
- simulation.py: camera_attach/frame/detach WS handlers
- esp32_worker.py: ctypes binding to velxio_push_camera_frame +
feature-detection fallback for older DLLs
- esp32_lib_manager.py: forward camera commands to the worker stdin
- esp-idf-template/main/CMakeLists.txt: esp32-camera headers added
via add_prebuilt_library + REQUIRES driver (resolves i2c_master_*
symbols). LED_BUILTIN=2 fallback for sketches that hardcode it.
Frontend (React/TS):
- EditorToolbar.tsx: ESP32-CAM (and the rest of the ESP32 family)
added to isQemuBoard list — Run button now starts the QEMU bridge
for these boards instead of falling through to the AVR path
- useWebcamFrames.ts: getUserMedia → OffscreenCanvas →
toBlob('image/jpeg') → base64 → WS at ~10 fps
- CameraToggle.tsx: header button with status colors + frame counter
- SimulatorCanvas.tsx: render CameraToggle for esp32-cam boards
- Esp32Bridge.ts: sendCameraAttach/Frame/Detach + chunked btoa
- useSimulatorStore.ts: diagnostic log on compileBoardProgram
- components-metadata.json: regen including esp32-cam component
Submodule pointer:
- wokwi-libs/qemu-lcgamboa → ff8eee0 (camera devices commit on
davidmonterocrespo24/qemu-lcgamboa branch picsimlab-esp32)
Investigation + tests in test/test-esp32-cam/:
- 13 autosearch markdown docs (overview, SOTA, OV2640 spec, DVP/I2S
spec, build blueprint, blockers resolved, descriptor walker fix)
- 5 sketches (camera_init, sccb_probe, dma_smoke, frame_roundtrip,
webcam_demo) + 8 live + WS regression tests
- README with the user-facing flow
.gitignore:
- libqemu-*.dll.{pre-camera,new,bak} (rollback points, regenerated)
- wokwi-libs/esp32-camera/ (clone consumed by arduino-esp32 path,
not part of this repo)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- Add version field to InstallLibraryRequest
- Add fallback and requested_version to InstallResponse
- Add DELETE /api/libraries/uninstall endpoint
- Enhance install_library() for versioned installs (LibName@version)
- Add semver validation and fallback logic
- Add uninstall_library() method
- Fix _parse_version() to reject non-numeric version parts
Frontend:
- Update installLibrary() with optional version parameter
- Add uninstallLibrary() and resolveLibraryVersion() helpers
- Add version selector dropdown in Library Manager
- Add UNINSTALL button for installed libraries
- Show fallback messages when requested version unavailable
- Add parseLibSpec() and version badges in InstallLibrariesModal
_create_idf_component was fixed but NOT _merge_arduino_libs_to_component.
The latter was still flattening library files, causing ArduinoJson compilation to fail
with 'src/ArduinoJson.h: No such file or directory'.
Changes:
- Replace flat copy with directory-structure-preserving copy
- Add exclusion logic for non-buildable directories (examples, tests, docs)
- Generate INCLUDE_DIRS from actual directory structure
- Track files by relative path to prevent name collisions
This ensures libraries with src/ layouts (ArduinoJson, etc.) compile correctly.
- Maintain original library layout (src/, utility/) instead of flattening files
- Add exclusion logic for non-buildable directories (examples, tests, docs, CI)
- Dynamically generate INCLUDE_DIRS from actual directory structure
- Add validation to ensure buildable source files exist before proceeding
- Support both flat and src-based library layouts
- Fix path separator normalization for cross-platform compatibility
This improves compatibility with complex Arduino libraries that rely on
specific directory structures and relative includes.
The project save/load pipeline only persisted a single `board_type`, so
multi-board workspaces silently lost every board except the active one
on save, and wires referencing the dropped boards' IDs orphaned to the
canvas corner on reload. An audit of the production backup found 74/306
projects (24%) with at least one orphaned wire and 174/301 non-trivial
projects whose code was still the default Blink template — strong signal
that users save once and never re-save.
Backend
- Add `boards_json` column on `projects` with idempotent ALTER TABLE in
the lifespan migration list.
- New `FileGroup` schema + `file_groups` array on
ProjectCreate/Update/Response. Legacy `files`/`code` kept for back-compat.
- `project_files.py` now uses `{pid}/{groupId}/{filename}` subdirs via
`read_groups`/`write_groups`. Legacy flat layouts are auto-promoted on
read; legacy single-list `files` only updates the active group, leaving
other boards' files intact.
- `_persist_files_from_body` honors file_groups → files → code priority.
Frontend
- `useSimulatorStore.addBoard` accepts an optional `explicitId` so
saved board IDs can be restored verbatim (wires reference IDs literally).
- New `loadProjectState({boards, fileGroups, components, wires,
activeBoardId})` action: tears down current boards, recreates from the
payload, restores file groups atomically, recalculates wire positions
on the next frame, and refreshes the Interconnect.
- `useEditorStore.replaceFileGroups` for atomic multi-group restore.
- `SaveProjectModal` and `ProjectByIdPage`/`ProjectPage` now go through
`buildSavePayload` / `buildLoadPayload` (handles pre-backfill projects
by synthesising a default board from `board_type`).
Auto-save (#useAutoSaveProject hook)
- 2.5s debounced silent PUT triggered ONLY when an authenticated user
has a `currentProject` with a UUID. State hash detects real changes
vs. UI-only churn; baseline is reset on project load so the just-loaded
state isn't immediately re-saved.
- `beforeunload` flush via `fetch keepalive: true` (supports PUT +
credentials, survives unload).
- Compact status indicator in `AppHeader` (idle/dirty/saving/saved/error).
Backfill script (one-off, idempotent)
- `backend/scripts/backfill_boards_2026_05.py` populates `boards_json`
for legacy projects. Heuristic per project, based on which board IDs
the wires reference:
Case A — wires only ref 'arduino-uno' but board_type ≠ uno:
rename id→board_type and rewrite wire endpoints.
Case B — single-board normal: keep verbatim.
Case C — multi-board: recreate one board per distinct ref, infer
kind by stripping trailing -N suffix.
Also moves any flat files into the active board's group subdir.
Stdlib-only, runs from host or `docker exec`.
Docker
- `Dockerfile.standalone` now copies `backend/scripts/` into the image
so the backfill is callable via `docker exec velxio-app python
/app/scripts/backfill_boards_2026_05.py --apply`.
Verified locally on the restored production backup (363 projects):
33 Case A, 316 Case B, 14 Case C, 135 wire endpoints renamed, 0 orphans.
Re-running the script after apply skips all 363 (idempotent).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Update esp32_lib_manager.py to dynamically set library extensions based on the platform (Linux, Windows, macOS).
- Update submodule references for qemu-lcgamboa and wokwi-elements.
- Add documentation on ESP32 Arduino runtime crashes related to cache disable during WiFi/BT initialization.
- Introduce regression tests for the ESP32-CAM blink issue, ensuring the user sketch matches the reported problem.
- Implement an IRAM-safe blink sketch to confirm the regression is due to the Arduino runtime.
- Create a comprehensive test suite to cover various layers of the simulation and compilation process.
- Implement UC8159cDecoder for handling 7-colour ACeP panels.
- Introduce painting functions for UC8159c frames in EPaperPart.
- Update EPaperPart to handle both SSD168x and UC8159c frame types.
- Add integration tests for EPaperPart and UC8159cDecoder.
- Create example sketch for 5.65" ACeP 7-colour panel.
- Enhance error handling in test cases for library dependencies.
- Introduced EPaperPanels.ts to define configurations for various ePaper panels including dimensions, refresh rates, and controller details.
- Implemented SSD168xDecoder.ts to handle the decoding of SPI commands for the SSD168x family of ePaper displays.
- Created EPaperPart.ts to manage the simulation of ePaper panels, integrating with the existing simulator architecture and handling events.
- Added example sketches for 2.13", 2.9", 4.2", and 7.5" ePaper displays, demonstrating basic functionality and text rendering.
- Ensured compatibility with AVR, RP2040, and ESP32 platforms, with appropriate pin configurations for each.
- Added tcp_nat.py to handle TCP NAT, implementing the three-way handshake, data flow, and connection state management.
- Introduced udp_nat.py for UDP NAT, managing chip-initiated datagrams and maintaining flow state.
- Created integration tests in test_picow_net_bridge.py to validate the functionality of the TCP and UDP NAT implementations, including ARP, DHCP, ICMP, and DNS interactions.
- Introduced SVG layout dimensions for Phase 1 (B/W mono) and Phase 2 (colour) ePaper panels, detailing active areas, bezels, and pin layouts.
- Developed a phased emulation plan outlining the architecture and deliverables for different panel types, including SSD168x and UC81xx.
- Created a canonical "Hello, World!" sketch for the 1.54" ePaper panel, ensuring compatibility across ESP32, Raspberry Pi Pico, and Arduino Uno.
- Implemented a pure Python SSD168x decoder to validate SPI command sets and framebuffers against specifications.
- Added tests for compiling the hello-world sketch across supported boards and for the SSD168x protocol to ensure correct framebuffer behavior.
- Implemented handshake tests to validate initial bus state and register responses.
- Created end-to-end tests for Pico W LED blinking using MicroPython firmware.
- Added SDPCM framing tests to ensure proper encoding and decoding of control frames.
- Developed IOCTL tests to verify command responses and state changes in the emulator.
- Established a full lifecycle test for WiFi operations, including scanning, connecting, and packet handling.
- Introduced TypeScript configuration for test files to ensure compatibility and strict type checking.
- Implemented `esp32_spi_chip_demo.ino` to demonstrate SPI communication with a 74HC595 shift register.
- Created `esp32_uart_chip_demo.ino` for UART loopback testing with ROT13 transformation.
- Added Python tests for compiling chips and sketches, ensuring valid WASM output and successful compilation for various board families.
- Developed end-to-end tests for ESP32 with custom chips using I2C and SPI, validating synchronous communication through the backend.
- Introduced GPIO bridge tests to verify serial communication and GPIO state changes.
- Ensured all tests validate the expected behavior of the custom chips and their interaction with the ESP32 firmware.
Track per-user, per-project, and per-board usage to inform pricing tier
decisions. Adds an admin dashboard with KPIs (DAU/WAU/MAU, totals,
success rate), time-series charts for compiles/runs, board family +
FQBN breakdowns, "board diversity" pie chart (key freemium signal),
top users/projects, and per-country breakdown via Cloudflare's
CF-IPCountry header. Admin can now also view private projects.
Backend:
- New UsageEvent table (append-only event log with user_id, project_id,
event_type, board_fqbn/family, country, error_kind, duration_ms)
- Aggregate counters on User (total_compiles/runs/errors, last_active,
signup_country, last_country) and Project (compile/run/update counts,
last_compiled/run timestamps) kept in sync by MetricsService for O(1)
dashboard reads
- 10 admin endpoints under /api/admin/metrics/{overview, timeseries,
boards, board-diversity, top-users, top-projects, countries,
users/{id}, projects/{id}}
- POST /api/metrics/run for client-side run telemetry
- Country detection via cf-ipcountry header (no DB / no API calls)
- Auto-migrations in lifespan for legacy DBs
Frontend:
- recharts-powered Dashboard tab with KPI cards and 4 charts
- New Boards tab with per-family + per-FQBN breakdown
- Country column with flag emoji on Users tab
- Top countries card on Dashboard
- compileCode now forwards project_id; Run button reports via WS
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Implement `ammeter-waveform.test.ts` to validate AC readings from a sine wave source.
- Create `capacitor-charge-transient.test.ts` to test the charging response of an RC circuit driven by a microcontroller pin.
- Introduce `esp32-rectifier-integration.test.ts` for testing rectifier behavior using QEMU and ESP32.
- Add helper functions in `esp32RectifierE2E.ts` for the rectifier test harness.
- Develop `voltmeter-waveform.test.ts` to ensure correct AC and DC readings from a sine wave source.
- Implement unit tests for waveform statistics in `waveform-stats.test.ts` to validate RMS, mean, peak, and interpolation functions.
- Create `waveformStats.ts` to provide statistical functions for time-domain waveform analysis.
- Implemented Bmp280Element as a custom web component for the BMP280 barometric sensor, including SVG representation and pin configuration.
- Created CircuitPreview component to render circuit thumbnails using SVGs of components, including support for various boards and components.
- Added a script to generate SVG files from wokwi-elements, ensuring proper formatting and structure for reliable rendering.
- Introduced a test HTML generation script to visualize component SVGs.
refactor: Update wiring examples for E32 OLED integration; correct pin mappings for VCC, GND, DATA, and CLK
test: Improve unit tests for ESPIDFCompiler; add scenarios for library resolution and CMake patching
chore: Mark subproject commits as dirty for wokwi-libs
- Implement tests for BMP280, DS1307, DS3231, I2CWriteSink, and MPU6050 slaves in test_i2c_slaves.py.
- Create test suite for Velxio MCP server tools in test_mcp_tools.py, covering Wokwi utilities and circuit management functions.
- Add tests for parsing WiFi and BLE serial output in test_wifi_status_parser.py, ensuring correct status events are captured.
David Montero Crespo
ZhadowValker