El bloque de velxio_board.cmake usaba board_fqbn, que vivia en compile() y no
en la firma de _compile_in_dir: parametro nuevo con default None, propagado en
los dos call sites, y guard para el camino sin fqbn (tests directos).
Tres piezas:
- _arduino_board_flags lee <id>.build.board y <id>.build.cdc_on_boot de
boards.txt (cache, mismo patron que _arduino_variant).
- Cada compile escribe velxio_board.cmake (incluido OPTIONAL por el
CMakeLists ANTES de project(), asi los defines llegan a TODOS los
componentes — que Serial sea el CDC se decide dentro del core Arduino):
ARDUINO_<BOARD> siempre que boards.txt lo nombre (los sketches de vendor
hacen #ifdef ARDUINO_XIAO_ESP32S3), y ARDUINO_USB_CDC_ON_BOOT=1 +
ARDUINO_USB_MODE=1 cuando cdc_on_boot=1. USB_MODE va FORZADO a 1 (HWCDC
por USB-Serial-JTAG, la opcion de menu USBMode=hwcdc): el motor modela esa
consola, no la pila OTG/TinyUSB. Ir por archivo y no por env hace el flag
dependencia de configure: CMake reconfigura al cambiar.
- CONFIG_ESP_CONSOLE_SECONDARY_NONE en el sdkconfig: sin ella el ROM/IDF
espejan cada byte de log al USB y el monitor unico del emulador (que
mezcla ambos streams) mostraria el log entero dos veces en builds CDC.
_normalize_options rellena arduinoVariant con _arduino_variant(board_fqbn),
que ya leia boards.txt; la plantilla ya consumia la clave. Sin esto todas las
placas S3 compilaban como esp32s3 generico: sin nombres Dx del XIAO y con
Serial en UART0, cuyo RX por defecto es GPIO44 = D7 del XIAO — un
pinMode(D7) tardio desinstala el driver y silencia los prints del loop
(medido en el banco con el ejemplo del Round Display).
Sin esta linea la variante se queda en el nombre del chip, asi que TODA placa
compila como dev-kit generico y variants/<VARIANTE>/pins_arduino.h nunca entra
en el include path: los nombres propios de la placa (D0..D10 en una XIAO, la
serigrafia de las M5) sencillamente no existen. Justo los sketches que la gente
copia de la wiki del fabricante.
El helper que deriva la variante del FQBN leyendo boards.txt ya estaba; faltaba
la plantilla que la consume. Sigue pendiente rellenar 'arduinoVariant' en
normalized para que surta efecto (ver F7.6 en velxio-prod).
Los dos ejemplos de camara (esp32cam-lcd-preview y esp32cam-webcam-demo) morian
con "esp_camera.h: No such file or directory".
No es que el ejemplo estuviera mal: es una regresion de la migracion. El arbol
viejo /opt/arduino-esp32 (2.x) traia un SDK precompilado con esp32-camera dentro,
en tools/sdk/<target>/include/esp32-camera/, para esp32, c3, s2 y s3. El
componente 3.x que usamos ahora NO lo trae, y su idf_component.yml tampoco
depende de el, asi que el header simplemente dejo de existir. Por eso funcionaba
antes y dejo de funcionar sin que nadie tocara el ejemplo.
Mezclar los headers de la 2.x en este build no es opcion — vienen con su SDK
precompilado y aqui compilamos contra IDF 5. Lo correcto en el mundo 3.x es
pedirle el componente al gestor: la plantilla no traia idf_component.yml, asi que
todo lo gestionado llegaba de rebote por el manifiesto de arduino-esp32, y lo que
el no arrastra hay que pedirlo explicitamente.
_detect_camera_usage() ve el include y _add_managed_components() escribe
main/idf_component.yml con espressif/esp32-camera. Verificado en el contenedor
sobre el mismo directorio que fallaba: el gestor descarga 2.1.7, el configure
termina con esp32-camera en la lista de componentes, y sketch.ino.cpp compila.
Arduino trata igual #include "Lib.h" que #include <Lib.h> para librerias, y
los ejemplos de los propios fabricantes usan la forma con comillas: los de
M5Stack para el Cardputer abren con #include "M5Cardputer.h". Al escanear solo
la forma con angulos, ese sketch no llegaba siquiera al resolutor de librerias y
moria con 'fatal error: M5Cardputer.h: No such file', mientras el mismo sketch
con angulos compilaba sin problema. Cualquiera que pegue un ejemplo oficial se
daba de bruces con esto.
Se pasan ademas los nombres de los ficheros del propio sketch, para que un
#include "Common.h" local siga siendo una cabecera del proyecto y no se
confunda con una libreria.
test/backend/unit/test_espidf_compiler.py cubre ambas formas, el espaciado, las
cabeceras internas de IDF y la cabecera propia del proyecto.
Las dos ramas habian divergido: master llevaba el modo lenguaje ESP-IDF puro
(#139) y v3.2 la ruta de compilacion IDF v5.5 para toda la familia ESP32 mas
los arreglos de venv/toolchain. Ambas tocaban espidf_compiler.py.
Los dos lados son ejes ORTOGONALES y se conservan enteros:
- use_idf5 / arduino_mode (v3.2): que arbol IDF usa el build (5.5 vs 4.4) y
si cabe Arduino-como-componente.
- pure_idf (master): el modo LENGUAJE que elige el usuario; sus ficheros son
las fuentes del componente main con su propio app_main().
Resolucion:
- _build_env acepta los tres. Un build IDF puro fuerza arduino_mode a falso:
la plantilla CMake mete el componente arduino-esp32 en cuanto existe
ARDUINO_ESP32_PATH, asi que dejarlo puesto compilaba el core de Arduino en
un build que no tiene sketch. Lo cazaron los tests de master.
- VELXIO_PURE_SKETCH solo con pure_idf, nunca con arduino_mode a falso a
secas: un target sin core arduino-esp32 sigue entregando un SKETCH al
traductor legacy y no debe tomar la rama del glob puro.
- La identidad del build-dir suma los dos tokens (|idf:N|ard:N y |lang:pure):
ningun par de esas combinaciones puede compartir un build/ configurado.
- La cadena de escritura de fuentes queda pure_idf -> arduino_mode -> legacy.
- sdkconfig: render de v3.2 (con target/use_idf5) mas el filtrado de simbolos
CONFIG_ARDUINO* de master cuando el build es puro.
test/backend/unit/test_espidf_compiler.py: los 7 tests que ya estaban rotos en
v3.2 (AttributeError: idf5_path, fixture sin actualizar desde que se anadio la
seleccion de IDF) vuelven a pasar.
Verificado: backend 293 pasan / 0 fallan (v3.2 traia 7 rotos); frontend 2268
pasan / 0 fallan en los dos shards.
Adds a third entry to the board language selector next to Arduino C++
and MicroPython: ESP-IDF. In this mode the user writes a plain ESP-IDF
project — app_main() entry point, FreeRTOS + driver APIs — and the
backend compiles it through the same ESP-IDF toolchain it already uses
for ESP32 Arduino sketches, just without the arduino-esp32 component.
Backend:
- CompileRequest.language ('espidf') threaded through the sync + async
compile paths and folded into the dedup job key (language='arduino'
and omitted hash identically so old clients keep dedupping).
- espidf_compiler: pure_idf flag. User files are written into main/
as-is (no Arduino.h wrap, no velxio_compat.h, Arduino library
resolution skipped), ARDUINO_ESP32_PATH is dropped from the build env
and VELXIO_PURE_SKETCH raised so the template CMake compiles the
user's own sources via a glob branch. Pure builds get their own
persistent build-dir variant through the eff_hash fold.
- QEMU WiFi compat for IDF-style code: esp_wifi.h/esp_wifi_init
detection sets has_wifi, and literal #define SSID/PASS plus
wifi_config_t designated initializers are normalized to the QEMU AP.
- CONFIG_ARDUINO_* lines are stripped from sdkconfig.defaults in pure
mode (the symbols don't exist without the arduino component).
Frontend:
- LanguageMode gains 'espidf'; BOARD_SUPPORTS_ESPIDF covers the ESP32
family (Xtensa, S3, C3). Toolbar shows the option only for those.
- Switching modes seeds a main.c blink skeleton (app_main + gpio
driver), mirroring the MicroPython main.py flow.
- compileCode sends language='espidf'; run/stop paths are unchanged
(the QEMU worker consumes the same merged flash image).
- New gallery example: esp32-idf-blink (LED + resistor on GPIO 2).
Tests: unit coverage for the build-env switch, IDF wifi normalization,
job-key variance, file-group seeding and the new example; verified
end-to-end in a container from the prod image (pure build produces a
bootable flash image; Arduino-mode build unchanged, same variant hash).
Ports the v5 compiler layer: /opt/esp-idf-v5 + arduino-esp32 3.x
discovery (IDF5_PATH / ARDUINO_ESP32_5_PATH overridable), per-compile IDF
selection routing the family to the v5 tree, esp32c6 target support, and
the docker-runtime hardening (version-matched python venv pinned on PATH,
toolchain ordering by the tree's tools.json, user-libs component REQUIRES
for the IDF components Arduino libraries include directly). The LED_BUILTIN
fallback in the project template becomes a whitelist so non-esp32 variants
(C6 first) never hit the redefinition trap. Includes the option tests.
The newest-env-wins heuristic pinned a 5.x env for a 4.4 tree — each
generation's dependency check only accepts its own requirement set. The
version is read from tools/cmake/version.cmake of the active IDF tree,
falling back to the newest env when unreadable.
Two failure modes seen on a fresh docker deploy where the final image's
system python differs from the one the IDF venv was created with:
- IDF's cmake derives the venv dir name from the SYSTEM python version
and fails with 'python doesn't exist' before configure; and when cmake
is invoked directly (not via idf.py) its PYTHON property falls back to
the bare 'python' from PATH. The Linux branch of _build_env now pins
IDF_PYTHON_ENV_PATH to the newest env under the tools root and puts its
bin first on PATH (mirroring what the Windows branch already did).
- When several toolchain generations share one tools root, glob order
decided which won the PATH race. Toolchain dirs are now ordered by the
versions the IDF tree declares in tools/tools.json.
drazzy.com (ATTinyCore index host) has had an expired TLS certificate
since 2026-06-22 and now 301-redirects http to https, defeating the
plain-http URL pinned to sidestep its TLS issues. Two failure modes:
1. A /root/.arduino15 volume from an older image can reference the index
in config while lacking the file; arduino-cli then fails instance
init outright, breaking EVERY compile, not just ATtiny (issue #254).
2. backend/Dockerfile chained update-index with &&, so any uncached
image build fails hard while the host is broken (exit 1 verified).
A stale index is harmless (ATTinyCore 1.4.1's platform archive and its
micronucleus 2.0a4 both download from github.com); a missing one is
fatal. So: vendor the index under backend/board-indexes/, copy it to
/opt/arduino15-seed in Dockerfile.standalone, and teach entrypoint.sh
to seed any missing package_*.json into /root/.arduino15 at boot,
healing stale volumes. backend/Dockerfile seeds the index directly and
makes update-index best-effort; core install lines stay strict.
Verified: removing the index reproduces the reporter's exact
'Error initializing instance' brick; after seeding, instance init
exits 0 with the host still broken.
_emit() wrote to stdout synchronously from QEMU callback context — the
iothread fires _on_uart_tx per UART byte, each becoming ~45 bytes of
JSON. When the parent's pipe reader stalled, the 64 KB pipe filled and
the write blocked INSIDE the QEMU iothread, freezing the entire guest.
Observed as an intermittent (~1 in 10) boot hang: serial output stops
right after the ROM log / rtcinit line — exactly where the accumulated
boot events cross the pipe capacity — and never recovers. Direct worker
harness runs (fast reader) never reproduced it; the browser/WS path did.
Route _emit() through a bounded queue drained by a dedicated writer
thread (opportunistic batching, one write per drain). Under extreme
backpressure events are dropped and counted on stderr — losing telemetry
beats freezing the emulated CPU. The shutdown path posts a sentinel and
joins the writer so crash/system events still flush before os._exit.
Verified on staging: 10/10 UI run/stop cycles + 8/8 direct harness boots
with identical serial latency (~0.95s to first app output).
_refresh_pin_pulls read a fixed 40 registers from get_internals(3), but
the exposed array is per-chip: classic muxgpios[40], ESP32-S3 49
(GPIO0..48). The fixed bound silently missed S3 pulls on GPIO40-48.
Read _GPIO_COUNT entries instead (set from the machine at startup).
Pairs with qemu-lcgamboa 547e989, which models the S3 IO_MUX pull bits
and exposes them via the get_internals override — together they make
INPUT_PULLUP buttons work on the emulated S3 exactly like the classic
ESP32 (verified on staging with GPIO4 and GPIO40).
The S3 has GPIO0..48; the identity pinmap length drives how many pins
picsimlab_wire_gpio connects to the host. Default 40 left GPIO40..48 unwired,
so digitalWrite/Read on those pins never reached the frontend. Build a 49-entry
pinmap when the machine is esp32s3 (mirrors the c3->22 special case). Requires
the GPIO-model widening in libqemu (esp32_gpio bank-1 8->17).
_idf_target had no S3 case, so every S3 FQBN (esp32s3 / XIAO_ESP32S3 /
nano_nora) compiled as plain esp32 (LX6) and could not boot the S3 QEMU
machine. Add _is_esp32s3 (covers all three FQBNs; nano_nora has no 's3'
token) + return 'esp32s3'; add the xtensa-esp32s3-elf toolchain to the
build PATH; place the S3 second-stage bootloader at flash offset 0x0
(like C3) instead of 0x1000 via 'is_c3 or idf_target==esp32s3' at the merge.
Requires the xtensa-esp32s3-elf toolchain in the image
(install.sh esp32,esp32c3,esp32s3) and a libqemu-xtensa with an
esp32s3(-picsimlab) machine.
Extend the spice-driven input path (already live for AVR/ESP32) to RP2040 and
STM32 so digitalRead() of an INPUT pin reflects the actual wiring: a pin tied
to a rail reads that rail, and an INPUT_PULLUP button-to-GND reads idle-HIGH /
pressed-LOW instead of floating or inverted.
RP2040 (rp2040js, frontend-only): the GPIO listener now splits input vs output
mode. Input pins report their pad pull (InputPullUp/Down) via setPinPull and
seed the pull's idle level (rp2040js does not auto-apply the pad pull to the
readable input register); the SPICE solve then overrides via connectDigital-
InputsToMcu when the net is actually sourced. Output pins drive as before.
spiceDrivenInputs = true.
STM32 (backend QEMU): the worker now forwards a new gpio_pull event (from the
libqemu-arm picsimlab_pull_pin callback) so the netlist stamps the matching
weak resistor; Stm32Bridge surfaces it, Stm32BridgeShim opts into
spiceDrivenInputs, and collectPinStates maps PA0/PC13 names to the linear pin
so the pull is read. STM32 outputs stay on the part layer (unchanged).
Event-driven parts with no SPICE model (rotary encoder, keypad) remain
protected by the existing sourcedNets gate in the connector.
Completes the internal-pull emulation for the common case (a button on an
RTC-capable GPIO like 4/15/25/... with INPUT_PULLUP):
- Backend reads the RTC_IO pad RUE/RDE bits via the new
get_internals(QEMU_INTERNAL_RTCIO) and emits gpio_pull for RTC pins, so
pull-up/down on those pads is finally visible (it lives in RTC_IO, not
IO_MUX). IO_MUX path still covers non-RTC pins.
- The digitalRead path is driven by seeding the GPIO input level, not by
SPICE. The part-level INPUT_PULLUP seed (BasicParts) is sent at attach,
before the multi-second QEMU boot finishes, so it is lost and the pin
reads LOW. makePinPullHandler now drives the pin to the pull's idle level
via sendPinEvent when the guest programs the pull (post-boot), so it
sticks. A real button press/release still overrides it.
INPUT_PULLUP / INPUT_PULLDOWN had no effect in simulation: the ESP32's
internal pull resistors live inside QEMU and were invisible to the SPICE
solver, so an input wired to a button-to-GND floated to 0 V and read LOW
even at idle. The canonical active-low button never worked.
Read the pull config straight out of the running guest: the IO_MUX
register (FUN_PU bit 8 / FUN_PD bit 7) is already exposed read-only via
qemu_picsimlab_get_internals(3), so no QEMU rebuild is needed. The worker
scans it on the 100 ms poll thread and emits gpio_pull; the bridge feeds
it to PinManager; the netlist stamps a weak 45k resistor to the rail so
idle inputs read the correct level. 45k matches the real internal pull
and is weak enough that any external driver/pull dominates.
Verified with ngspice: idle ~3.3 V (HIGH), pressed ~0 V (LOW).
- boot_images manifest: bump arm64 rootfs (gpiozero/colorzero baked in,
hostname applied at boot, reworded MOTD)
- RaspberryPi3Bridge: onBooted shell-ready detector + sendAndWaitForPrompt
flow control (resets on disconnect)
- RaspberryPiWorkspace: distinct Booting overlay + piBooted-driven status,
inline SVG icons replacing emoji glyphs
- SerialMonitor: strip CSI/DSR escapes so the dumb console no longer shows
a literal [6n next to the prompt
- VirtualFileSystem: upload auto-starts the Pi and waits for the shell, then
flow-controls each command (no more dropped lines on large files)
- i18n: bootingTitle/bootingNote + reworded offlineNote2 across 9 locales
Move the CYW43439 (Pico W) WiFi emulation out of the open-source tree so it
can ship as a paid feature in a private overlay. OSS keeps a plain Pico W
(no WiFi); the overlay registers the cyw43 protocol + backend network stack
at runtime via generic seams.
Frontend:
- Add simulation/PioPeripheral.ts: a generic "PIO bus peripheral" seam
(feedWord / inDiscardableWriteData / resetFraming / hostWakeLevel /
onHostWake / onSimulationStart). No factory is installed in OSS, so
createPioPeripheral() returns null and a Pico W simulates as a plain Pico.
- RP2040Simulator: keep the fragile PIO-FIFO plumbing (it must re-run after
loadMicroPython swaps the chip) but drive it through PioPeripheral instead
of an inlined cyw43 import (attachCyw43 -> attachPioPeripheral, etc.).
- useSimulatorStore: generic attach/detach + setBoardWifiStatus; drop the
cyw43 bridge map.
- MicroPythonLoader: add registerFirmwareVariant() so an overlay can add the
RPI_PICO_W build; remove the OSS pico-w config + bundled .uf2.
- Delete simulation/cyw43/ (moved to the overlay).
Backend:
- core/hooks.py: add generic register_ws_sim_handler / dispatch_ws_sim_message
and register_gateway_proxy / dispatch_gateway_proxy seams.
- simulation.py: route start_picow / stop_picow / picow_packet_out through the
ws_sim_handler hook (the overlay handles + gates them).
- iot_gateway.py: resolve the Pico W gateway through the gateway_proxy hook.
- Delete services/picow_net/ + picow_net_bridge.py (moved to the overlay).
Tests: move the cyw43/picow suites to the overlay; update RP2040Simulator
mock stubs to attachPioPeripheral.
Two real-world failures hit the Pico W gateway that the headless e2e
(node fetch, tiny request, fast timing) didn't surface:
- A browser sends KILOBYTES of headers (cookies, User-Agent, sec-*).
Forwarded verbatim, the request overran the chip's small recv()
(e.g. recv(1024)); lwIP then RST the connection on close-with-unread-
data, crashing blocking-socket sketches with ECONNRESET. Now we forward
a MINIMAL request (method, path, Host, Connection: close, and
Content-Type/Length for bodies) — nothing a tiny server can choke on.
- After a gateway request completed we dropped the connection immediately,
so a late chip segment (retransmitted FIN / trailing ACK) no longer
matched and fell through to the chip-initiated NAT, which RST it. Add a
short TIME_WAIT: keep the connection briefly and re-ACK late segments so
the chip closes cleanly, never RSTing it.
ESP32 web-server examples are reachable from the browser via
/api/gateway/<client_id>/ (QEMU slirp hostfwd). The Pico W server lives
in the browser-side lwIP, so there was no inbound path: visiting the
chip's IP did nothing.
Add the mirror of tcp_nat.py: tcp_inbound.TcpInbound originates a TCP
connection INTO the chip over the WebSocket bridge (SYN -> SYN+ACK ->
ACK -> request -> response -> FIN), so the backend can fetch a page the
sketch serves on 10.13.37.42:80 and hand it back to the browser.
- bridge.py routes chip TCP segments addressed to a gateway-opened
connection to TcpInbound (before the chip-initiated NAT, which would
RST them); exposes http_into_chip() + ensure_chip_mac() (primes the
chip's gateway ARP).
- iot_gateway.py: same /api/gateway/<client_id>/ route now falls through
to the Pico W bridge when there's no ESP32 instance, builds a raw
HTTP/1.1 request, and parses the chip's response. Same plan gate, same
URL shape — the browser sees no difference between ESP32 and Pico W.
Validated end to end (real RP2040 emulator serving an HTTP page ->
gateway returns it) plus 6 unit tests for the TCP state machine,
response parsing and ARP priming.
Add a working microSD card part backed by a FAT16 image, following the
Wokwi storage model: the project's own workspace files are auto-copied
onto the card (free), and an optional "SD Card" panel uploads extra
files (gated as a paid feature by the velxio.dev overlay; OSS default
allows it).
Frontend (in-browser AVR / RP2040):
- ProtocolParts.ts: rewrite the microsd-card part from a handshake stub
into a real SD-over-SPI device (reply-first Ncr timing, SDSC byte
addressing, single/multi-block read+write, CSD/CID, full CMD set).
- utils/fatImage.ts: dependency-free FAT16 super-floppy builder (8.3 + LFN).
- utils/sdCardFiles.ts: assemble the card image from workspace files plus
uploaded files; base64 helpers.
- components/simulator/SdCardPanel.tsx + ComponentPropertyDialog: upload UI.
- DynamicComponent + useSimulatorStore: build and inject the image on run.
- lib/proSdCardGate.ts: overlay-installable gate for the upload action.
- data/examples-storage-microsd.ts: Arduino Uno + ESP32 gallery examples.
Backend (ESP32 via QEMU):
- services/esp32_sd_slave.py: synchronous SD-over-SPI slave (Python port of
the browser part) with a sparse backing store, idle-state R1 tracking and
real CRC16 on data blocks when the host enables CRC (CMD59) -- both
required by ESP-IDF's sdspi driver.
- esp32_worker.py: route SPI bytes to the slave (returns MISO synchronously)
and feed write-only bulk transfers.
- esp32_lib_manager.py + routes/simulation.py: forward the FAT image
(sd_card.image_b64) from the start config into the worker.
Tested:
- frontend: protocol-parts, fat-image, sd-card-gate and microsd-real-firmware
(real Arduino SD.h on avr8js) -- 86 passing.
- backend: test_esp32_sd_slave (10) covering the ESP-IDF init sequence and
CRC16; validated end to end by running a real SD.h sketch in libqemu-xtensa
(mount, directory listing, read and write-readback).
arduino-esp32's WiFiClientSecure/ssl_client.cpp wraps its ENTIRE body
(start_ssl_client, ssl_init, send_ssl_data, ...) in
#if !defined(MBEDTLS_KEY_EXCHANGE_SOME_PSK_ENABLED) ... #else <body> #endif
ESP-IDF's mbedtls defaults the PSK key-exchange modes OFF, so the object
compiled empty and any sketch using WiFiClientSecure — including
HTTPClient.begin(url), which links the secure client even for http:// —
failed to link with "undefined reference to start_ssl_client". Commenting
out begin() let the optimizer drop the unused client, which is why it
"compiled when commented".
- sdkconfig.defaults.in: enable the PSK key-exchange ciphersuites
(CONFIG_MBEDTLS_PSK_MODES + the four KEY_EXCHANGE_*PSK), matching
arduino-esp32's own sdkconfig.
- espidf_compiler: ESP-IDF only seeds sdkconfig from sdkconfig.defaults when
sdkconfig is ABSENT. Persistent build dirs live in the build volume and
keep a stale sdkconfig across image rebuilds, so the new CONFIG_* would
never reach kconfig. Drop the generated sdkconfig when the rendered
defaults change so it re-seeds on configure.
- test: assert the rendered sdkconfig enables PSK.
Verified end-to-end: the reported WiFi+HTTPClient sketch now compiles to a
1.1 MB binary (was a hard link error before).
list_installed_libraries() now honors VELXIO_FALLBACK_SKETCHBOOK (pro overlay) so
GET /api/libraries/list enumerates the content-addressed cache instead of the
shared global dir — the Installed tab survives the global volume's retirement
(audit finding #3). Unset (OSS self-host) -> default sketchbook, unchanged.
The last paths that read the shared global /root/Arduino/libraries: a compile
with NO manifest (libraries=null — any from-scratch/anon sketch; the manifest is
never auto-derived from #includes) and the incomplete-manifest scan-all retry
(which re-enters compile unscoped). Both fell through to the global dir,
bypassing the cache entirely (a cached lib still failed when global was gone).
Now, when no scope is materialized, point the library search at the cache: the
cache root is itself a valid Arduino libraries dir (each <name@ver-sha> child is
a library), exposed via env (pro overlay sets them):
- arduino-cli: ARDUINO_DIRECTORIES_USER = VELXIO_FALLBACK_SKETCHBOOK (whose
libraries/ -> cache root) when scope_dir is None.
- ESP-IDF: _find_arduino_libraries_dir() prefers VELXIO_FALLBACK_LIBRARIES_DIR.
Unset (OSS self-host) -> legacy global, unchanged.
Also strip a trailing @version from manifest names (_bare_lib_names): norm_name
fused 'ArduinoJson@6.21.5' -> 'arduinojson6215' (cache miss -> global scan-all);
the per-board boards_json manifest is the path that still carries @version.
The compile scope resolved a project OWNER's per-user custom libraries for ANY
project_id a requester supplied, with no visibility check — so a requester who
knew a victim's PRIVATE project_id + custom-lib name could compile a binary
against the victim's private uploaded library. Replace the ungated
get_project_owner hook with resolve_compile_owner(project_id, requester_id),
which returns the owner ONLY when the requester IS the owner OR the project is
shareable (public/unlisted); otherwise None -> the caller falls back to the
requester's OWN store. Fails closed on any error.
Also gate the server-side manifest fallback by the same rule (symmetry): a
private project's declared library NAMES are no longer read into a non-owner's
compile scope. The owner's own compile and shared/embed compiles of public/
unlisted projects keep their backend-authoritative scope unchanged.
Empirically, arduino-cli's --libraries ADDS to the search path — the global
sketchbook is STILL scanned, so the prior commit did NOT isolate reads from the
shared global volume. Point ARDUINO_DIRECTORIES_USER at the scratch sketchbook
instead (its <scratch>/libraries becomes the ONLY user-library dir); cores +
board-manager URLs live in the DATA dir and are untouched.
In-container functest (AVR uno): a cache-only lib compiles via the scope (never
in global); a global-only lib NOT in the manifest is INVISIBLE to the scoped
compile and only recovers via the scan-all retry (manifest_incomplete=True) —
proving the global volume is no longer scanned for a manifest-scoped build.
arduino-cli compiles now read libraries from a per-compile --libraries dir of
symlinks materialized by the pro overlay (owner store -> cache -> legacy), via
the existing materialize_library_scope hook, instead of scanning the shared
mutable global volume. --libraries only overrides the USER library search path,
so cores + board-manager URLs (RP2040 earlephilhower, ATTinyCore) are untouched.
Mirrors the ESP-IDF P2.1e graceful fallback: an incomplete manifest (a needed or
transitive lib not declared) makes the scoped compile miss a header; we retry
ONCE scan-all (no --libraries) and surface manifest_incomplete, so a partial
manifest degrades to legacy behavior instead of hard-failing.
Dedup correctness: the manifest + owner are resolved ONCE (shared
_resolve_compile_scope) and folded into the /compile/start dedup key AND threaded
into the build, so two owners with identical sketch+board but different custom
libs never coalesce to one another's job, and the key never diverges from the
bytes the build uses. Owner folded only when a manifest applies (index-only
compiles keep cross-owner dedup).
New OSS warm_library hook; /api/libraries/install now calls it (with the
requester id for the anon policy) instead of mutating the shared global
libraries volume — so that volume stops growing and can be retired (P2.5).
Falls back to the legacy arduino-cli global install when no overlay is loaded
(OSS self-host parity).
- compile.py: owner_id = project owner ELSE the requester (so an unsaved
compile resolves the libs the user just uploaded, which are their own);
threaded requester_id into _run_compile from both call sites.
- LibraryManagerModal: on a custom .zip upload, auto-add the lib to the active
board's velxio.json + show the Project tab, so the compile resolves it via the
owner per-user path (the upload now lands in the per-user store, not the
shared dir, so it must be declared to be found).
So a scoped compile can resolve the project OWNER's per-user custom libraries
(not the requester's) — a shared/embed/anon compile of someone else's project
still finds that owner's uploaded libs.
- core/hooks.py: new get_project_owner hook; materialize_library_scope gains an
opaque owner_id param (no-op default unchanged).
- espidf_compiler.compile/_attempt: thread owner_id to the materializer.
- compile.py: resolve owner via get_project_owner(project_id), pass to compile.
Additive: the OSS image (no overlay) ignores owner_id; index libs still resolve
from the cache. Foundation for per-user custom-lib storage (P2.2a write side).
A scoped ESP32 compile can now resolve libraries from a per-compile directory
provided by an overlay (the manifest's libs symlinked from a content-addressed
cache, with a legacy-dir fallback) instead of the single shared global volume.
- core/hooks.py: register_materialize_library_scope / materialize_library_scope
(no-op default -> None, so the OSS image keeps its single scan-all dir).
- espidf_compiler: _attempt(allowed) calls the hook, folds the returned content
token into the build-variant eff_hash (a content change gets a clean build
dir), passes libraries_dir to _compile_in_dir (arduino_libs = libraries_dir or
_find_arduino_libraries_dir()), and removes the throwaway dir after. The
graceful scan-all fallback (allowed=None) keeps using the default dir, so the
worst case of any materializer failure is fall-back-to-legacy (no break).
Library manifests are now PER-BOARD (each board carries its own velxio.json),
so two boards in one project can use different (even conflicting) libraries
without clashing — the multi-board extension of the no-clash guarantee.
- board.libraries on BoardInstance + serialisableBoard: rides in boards_json,
so it round-trips, dirty-checks, autosaves and restores natively. This also
removes the load-restore hacks (useLibraryManifestStore + applyProjectManifest
deleted): the manifest is plain board state.
- loadProjectState now restores per-board boardOptions/spiffsFiles/libraries
(it previously dropped them).
- EditorToolbar single + compile-all send the COMPILING board's libraries.
- Backend compile.py prefers the client's per-board request.libraries; the
project-level libraries_json (now the union of all boards) is the fallback.
- buildLoadPayload migrates pre-per-board projects: seed each board with the
project union so they keep compiling scoped.
- Library Manager 'In project' tab edits the ACTIVE board's velxio.json (shows
the board name) and the add field is now an autocomplete (installed libs +
index search) so users pick from a list instead of typing names.
Deletes useLibraryManifestStore.ts + applyProjectManifest.ts.
Adds the get_project_libraries hook: the compile route reads a saved project's
declared library manifest (by project_id) and uses it as the ESP-IDF resolution
scope, preferring it over the client-sent manifest. So a saved project always
compiles against only its own declared libraries — never another user's, or
another project's, stray install in the shared dir — authoritatively from the
server, independent of frontend wiring. Client-sent manifest still used for
unsaved examples; None/empty → legacy scan-all. Overlay fills the hook in
register_pro; OSS default is no-op (None).
Per-variant build dirs fixed the cross-compile staleness, but the FIRST build of
a cold variant can still occasionally hit ESP-IDF nested-build flakiness (cmake /
bootloader / managed-components / sdkconfig). These are infrastructure failures,
never user code, and clear on a retry once the variant dir is warmer. Retry the
attempt once when the failure matches infrastructure markers (not a user-sketch
or missing-library error). Cheap via ccache + ninja incremental.
Replaces the wipe-on-change approach (which left ESP-IDF's nested bootloader /
managed-components build in a broken state under rapid reconfigure -
intermittent 'managed_components_list.temp.cmake: No such file'). Each distinct
configuration (board options x resolved library set, via the variant key the
caller already computes) now gets its OWN persistent project dir with its OWN
build/, never wiped or reconfigured for a different config:
- same config -> same dir -> warm ninja incremental + ccache (fast iterate);
- different config -> different dir -> isolated, consistent, no staleness,
no nested-build breakage;
- the scoped vs scan-all fallback attempts land in different dirs, so the
double-compile no longer corrupts a shared build/.
Variant dirs are LRU-bounded (_MAX_BUILD_VARIANTS per target); the global ccache
warms a fresh/evicted variant in seconds. Cleans up the old single-project
layout on first run. Fixes the cross-compile staleness for legacy AND manifest
compiles, and the fallback regression.
Supersedes the mid-_compile_in_dir build/ wipe (161deb9): wiping build/ AFTER
materializing libs but right before cmake left ESP-IDF's config half-regenerated
during the fallback's scoped->scan-all double-compile, intermittently failing
with 'sdkconfig.h: No such file'.
Instead fold the effective library set (the manifest, else the sketch's
non-core external includes) into the per-attempt build-dir hash, so a changed
lib set — or the scan-all fallback after a scoped attempt — resets the
persistent build/ at _prepare_persistent_project_dir time (before any cmake).
That is the existing, well-tested early-wipe path, so the configure is always
clean. Same lib set across compiles keeps the warm ccache/ninja cache; core-only
sketches share one dir (core headers filtered out of the token) so they never
trigger a spurious wipe.
Fixes both the original cross-compile staleness (intermittent cmake-configure
failures + stale-object false positives) and the fallback regression.
The persistent per-target build/ caches ESP-IDF's cmake configuration, ninja's
incremental graph and ccache-backed objects, all assuming a stable component
set. When consecutive compiles on the same dir have a DIFFERENT resolved
user_libs set (a different project/user, or a different library manifest) that
cache is inconsistent and produces two real failures:
- cmake reconfigure intermittently fails ('cmake configure failed') even
though each manifest compiles fine on a clean dir;
- ninja/ccache reuse a previous compile's objects/headers, letting a
now-absent library slip through as a false-positive success against a lib
the current sketch/manifest no longer includes.
Fingerprint the materialized user_libs/ (sorted relative paths + sizes) and,
when it changes vs the last compile on this dir, wipe build/ to force a clean
configure. ccache (enabled) refills the objects so the rebuild stays cheap.
No-op on the ephemeral path and the first compile. Fixes both symptoms; will be
superseded by the fully ephemeral per-compile workspace (P1).
A manifest-scoped compile that fails because a header isn't in the manifest
(an undeclared/transitive dependency) now retries once with scan-all, so a
project with an incomplete manifest still compiles instead of regressing.
The response reports manifest_incomplete=true and
manifest_suggested_libraries={header: [candidate lib names]} so the manifest
can be auto-completed (P2.4) or the user prompted to add the missing library.
This de-risks turning on manifest sending (P2.3): an incomplete example/project
manifest can never break a build that worked before.
- compile(): _attempt(allowed) helper; retry scan-all on missing-lib failure.
- _missing_library_headers / _suggest_libraries_for_headers helpers.
- CompileResponse.manifest_incomplete + manifest_suggested_libraries.
P2.0 first cut took the first-alphabetical lib providing a header and then
checked manifest membership. When several installed libs ship the same header
(e.g. DHT118266, DHT_sensor_library, servodht11 all have DHT.h), the stray
first-match got rejected and the header was dropped even though the declared
lib provides it.
_find_manifest_library_for_header: when a manifest is supplied, pick the first
DECLARED library that provides the header. This both selects the right lib and
excludes undeclared ones. No manifest = legacy first-match.
Test strengthened with a stray same-header lib that sorts first.
When a compile supplies a 'libraries' manifest, _resolve_library_components
merges a USER-installed library only if it's declared in that set. A sketch
therefore never picks up an unrelated library from the shared dir (another
user's install, or a same-named clash) — the manifest is the resolution scope.
- _resolve_library_components(allowed_libraries): gate user-lib merges on
manifest membership; match by folder name OR library.properties name=,
normalised (display name vs on-disk folder differ by separators/case).
Core/bundled libs are never gated. None = legacy scan-all (unchanged).
- Threaded through compile() -> _compile_in_dir.
- compile.py: CompileRequest.libraries; folded into the async dedup _job_key
so a different manifest doesn't dedup to a job built with another.
Opt-in: omitting 'libraries' preserves current behaviour exactly.
Regression: test/backend/unit/test_espidf_core_first.py::TestManifestScope
A user library that ships a core-named header (e.g. WiFiEspAT/src/WiFi.h)
could shadow the arduino-esp32 core during ESP-IDF library resolution.
WiFiEspAT shadowing WiFi.h pulled EspAtDrv.cpp into the build, whose
const char OK[]/STATUS[] collide with ESP-IDF's enum STATUS in
rom/ets_sys.h, breaking every ESP32 sketch that #include <WiFi.h>.
_resolve_library_components now:
- skips a header entirely when the arduino-esp32 core provides it
(computed set from cores/ + libraries/, cached), so a user lib can
never shadow WiFi.h/Wire.h/SPI.h/WebServer.h/...
- skips a resolved user lib whose library.properties architectures=
excludes esp32/*.
Regression: test/backend/unit/test_espidf_core_first.py
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 7.5" 800x480 dashboard (GxEPD2_750_T7) rendered blank: it is a UC8179 /
GD7965 controller, but the panel config claimed controllerFamily 'ssd168x',
so the SSD168x decoder (which only reads 0x24/0x26/0x44/0x45) ignored its
0x10/0x13 DTM stream.
- Add a Uc8179 decoder (worker Uc8179EpaperSlave + browser Uc8179Decoder).
UC8179 is the same UltraChip command family as the UC8159c (0x10/0x13 DTM,
0x12 refresh) but mono (1 bit/px). GxEPD2 writes the visible image to 0x13
(DTM2 "current"; 0x10 is the ignored "previous"), framed by 0x91/0x90
(partial window, pixel coords MSB-first)/0x13 data/0x92. Data lands at
absolute pixel coords inside the window, so compose is just the RAM. The
Frame reuses the SSD168x palette (0=black, 1=white) so paintFrame renders it.
- EPaperPanels.ts: add the 'uc8179' family and point epaper-7in5-bw at it.
EPaperPart.ts + esp32_worker.py dispatch 'uc8179' to the new decoder.
- Fix the BUSY polarity: UC8179 (like the UC8159c) idles BUSY HIGH, not LOW.
The worker seeded BUSY LOW for every non-uc8159c panel, so GxEPD2_750_T7's
_PowerOn()/_InitDisplay() busy-wait timed out (~10 s, "Busy Timeout!") on
every refresh. Now _PowerOn returns in ~129 us.
- esp32_worker.py: the runtime sensor_attach epaper path still emitted the
epaper_update payload nested under 'data' (the old double-wrap bug); emit
it flat like the init path.
The 5.65" ACeP UC8159c example already rendered (it has its own decoder and
got the WS-plumbing fix); verified the 7 colour bars are correct.
The 2.9" tri-colour ESP32 alert badge rendered the red ALERT pill as white:
the red plane (0x26) was received but landed out of bounds and was dropped.
GxEPD2_3C writes the 0x24 (black) plane then the 0x26 (red) plane WITHOUT
re-seeking the RAM address counter between them — it relies on the SSD168x
counter wrapping back to the window start after the last byte of the window.
Our decoder advanced Y past the window end instead of wrapping, so every
0x26 byte hit y >= rows and was discarded (red_ram stayed all-init).
Mirror the hardware: when the X cursor wraps at the end of a row, advance Y
with a wrap at the active window boundary (yrange), honouring the data-entry
Y direction. Applied identically to the worker slave, the browser decoder,
and the Python golden reference so the three stay in lockstep. No regression
on the mono panels (their counter is re-seeked per plane, so the wrap is a
no-op for them); verified the tri-colour pill now renders red and the 2.9"
weather / 2.13" clock / 1.54" hello panels are unchanged.
ePaper panels rendered rotated/misaligned on AVR and RP2040 (e.g. the 2.13"
Pico clock came out sideways and clipped). The ESP32 worker decoder was just
taught to compose in the controller's native RAM geometry and rotate to the
display orientation, but the browser-side SSD168xDecoder (used by AVR/RP2040)
still composed at display dims with no rotation, so the two diverged.
- SSD168xDecoder.ts: port the worker's native-window compose + rotation.
* Size RAM to the longer side both ways so a rotated native layout
(128x296 behind a 296x128 panel) isn't truncated.
* Compose in the active RAM window, then rotate via the inverse of
Adafruit_GFX setRotation(1). Detect orientation by BYTE width so a
non-multiple-of-8 native width (the 2.13" panel is 122 px) is handled.
* Track the UNION of windows per frame: paged drivers (GxEPD2 page height
< panel) set one partial window per page, so compose must use the full
native area, not just the last page's strip. Fixes the all-white render
on paged panels (1.54" Uno, 4.2" Pico, 7.5" ESP32).
* Add an isBwr option: B/W panels treat 0x26 as a 2nd mono plane (white
only if both planes white), tri-colour panels keep red-wins.
* Default the active window to display geometry; the firmware overrides it.
- EPaperPart.ts: pass isBwr = cfg.palette === 'bwr' to the decoder.
- esp32_spi_slaves.py / esp32_worker.py: mirror the byte-aware rotation +
window-union in the worker, and derive is_bwr from panel_kind on the
runtime sensor_attach path too (fixes the tri-colour ESP32 alert badge).
- test_epaper/ssd168x_decoder.py: re-port the golden reference to match
(keeps the 3-way TS/Python/worker identity invariant). Tests updated to
construct tri-colour cases with is_bwr/palette='bwr'.
- examples-displays-epaper.ts: the Pico VCC wire referenced '3V3(OUT)',
which the velxio-pi-pico-w element doesn't expose (it has '3V3'), so the
wire snapped to the board corner. Use '3V3'.
ESP32 ePaper examples (e.g. epaper-2in9-esp32-weather) rendered as a blank
white panel. Two bugs, both above the SPI layer:
1. The worker's epaper_update event nested its payload under 'data', unlike
every other (flat) worker event. The backend qemu_callback re-wraps the
post-'type' payload under 'data', so the frontend received
msg.data.data.component_id (undefined) and EPaperPart bailed on
id !== componentId, so paintFrame/putImageData never ran. Emit it flat.
2. Ssd168xEpaperSlave was sized to the display dims (296x128), but GxEPD2
with setRotation(1) writes the controller's NATIVE RAM (128x296). The
_y < height bound dropped rows 128-295 (half the image) and compose
never rotated. Size RAM to the longer side, compose in the native
active-window geometry (0x44/0x45), then rotate to the display
orientation (inverse of Adafruit_GFX rotation 1). Add is_bwr (from
panel_kind): B/W panels init the 0x26 plane white and compose
white-only-if-both (GDEY029T94 mirrors the image into 0x26); tri-colour
panels keep red init 0x00 and red-wins.
Worker side of the libqemu picsimlab_spi_event_batch / CS-gating change:
- _on_spi_batch(): replay a whole SPI transfer in bulk (custom-chip runtime,
then ePaper feed, then the spi_batch buffer) instead of one _on_spi_event per
byte. Registered as a trailing _SPI_BATCH field of _CallbacksT.
- _sync_cs_events(): disable SPI chip-select callbacks for pure-display sims,
enable them when an ePaper / custom-chip SPI slave is registered (no-op on
older libqemu without qemu_picsimlab_enable_spi_cs_events).
- _on_pin_change(): flush the SPI batch before each gpio_change so the byte
stream stays ordered against the DC pin now that CS no longer triggers the
flush.
Backward compatible: an older libqemu never calls the batch callback or the CS
setter, so it just keeps the per-byte path. esp32-doom: 0.04 -> ~1.0-1.5 FPS
wall-clock (~26-37x), render verified correct.
SDCC's z80 crt0 puts the reset vector at 0x0000 (jp init) and the init stub
(set SP, call _main) at an absolute .org 0x100. Passing `--code-loc 0x100`
relocated the _CODE segment on top of that init stub, so on reset the CPU
jumped into __clock/_exit (rst 0x08 then ret with a garbage stack) and
derailed into NOP land before ever reaching main — every Z80 C program ran
but drove nothing (z80-led-chaser-c compiled yet the LEDs never moved).
Verified via a standalone chip-WASM harness: with the flag the chaser does 0
LED writes; without it, it walks the bit (8 writes). Pairs with the z80-cpu
RAM map now covering 0x8000-0xFFFF so the crt0's SP=0 stack is real RAM.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
David Montero Crespo