Cada guest es un proceso QEMU con su propia RAM (1-2 GB segun placa) y
sus hilos de vCPU, asi que el limite lo pone la maquina, no el codigo.
No habia ningun tope: el usuario N simplemente empujaba la caja a swap y
la sesion de TODOS se volvia lenta, que es peor que decirle al usuario N
que espere un minuto.
VELXIO_PI_MAX_INSTANCES (6) guests simultaneos en la maquina
VELXIO_PI_MAX_PER_OWNER (2) guests por persona
El "owner" es el hash de la cookie de sesion: sirve solo para contar, no
se guarda ni se lee de vuelta, y cae al host del cliente cuando no hay
cookie (sidecar de escritorio, tests). Sin identidad solo aplica el tope
global.
El rechazo llega como un mensaje que dice que hacer ("prueba en un
minuto" / "para una de tus sesiones"), no como un fallo mudo, y se
comprueba ANTES de lanzar el proceso.
Auditoria de "la Pi tiene todo lo de la placa real" con cuatro huecos
encontrados y cerrados:
1) GPIO de entrada en modo Linux: GPIO_IN respondia VAL 0 fijo (stub de
la fase 2), asi que GPIO.input() leia 0 eternamente aunque el canvas
empujara el nivel. El backend guarda ahora el ultimo nivel por pin
(set_pin_state lo escribe) y GPIO_IN contesta de ahi. Los flancos
(SET) siguen llegando al guest como antes.
2) UART del header hacia otra placa: el shim del rootfs ya hablaba
`UART <port> TX <hex>` / RX_REQ, pero sin modelo de esclavo el
backend tragaba los bytes. Ahora TX sin esclavo se emite al canvas
(uart_tx) y RX_REQ sin esclavo drena la cola que llena pi_uart_rx —
el mismo protocolo de siempre, sin ops nuevas.
3) El escaner de esclavos I2C/SPI/UART estaba doblemente muerto:
clasificaba por numero fisico de pin ('3','5','19'...) cuando el
elemento expone GPIOxx, y su unico llamador era RaspberryPiWorkspace,
que el terminal unificado reemplazo. Acepta ambos nombres y corre en
onBooted del store.
4) boardPinToNumber solo mapeaba los pines de la 3/4/5; la Zero, 1B+ y
2B (mismo header de 40 pines, mismo elemento) se quedaban sin mapa.
La placa QEMU-Linux tiene DOS flujos serie y hasta ahora el cableado
usaba el equivocado: el enrutado entregaba los bytes del vecino a la
consola (el shell) y sacaba al cable la cháchara del arranque. El
header, que es lo que el usuario cablea, no existia.
Ahora el canal de protocolo lleva dos ops nuevas:
UARTTX <b64> el guest transmitio por el header -> al canvas
UARTRX el guest pregunta que le llego -> UART_RXQ <b64>
El backend guarda una cola por instancia (acotada a 64 KB, que un script
que no lee nunca no la haga crecer) y el websocket acepta `pi_uart_rx`
con los bytes que el vecino manda. En el frontend el bridge gana
onUartTx / sendUartBytes y el Interconnect engancha ESE flujo en vez de
la consola para las placas Pi.
Con esto el mismo script -- import serial, escribir, dormir, leer --
funciona en los dos motores.
Guests stay on '-nic none' unless a profile opts in, and opting in does
NOT mean internet: the NIC is user,restrict=on (no route out, no route
to the host LAN) with one guestfwd to whatever command the overlay
configures — a filtering proxy in practice. Keeps 'user code never gets
a raw socket outside' true by construction.
A forgotten tab pinned a QEMU process and its RAM for as long as the
browser stayed open. Instances now shut themselves down after
MAX_SESSION_SECONDS (2 h default, env-tunable) and say so on the serial
line instead of vanishing.
A profile's extra_drive is the same file for everyone; an overlay may
need a disk built for THIS session (what the project declared). New
seam set_pi_extra_drive_resolver(fn) receives the client id, the board
and the start_pi payload and returns raw images, mounted read-only after
the profile's own. The WS route forwards msg_data and the bridge gained
startPayload so a client can declare it. Generic: no package manager,
no OS knowledge in the OSS tree.
- profile key extra_drive: optional read-only second virtio-blk so an
overlay can ship guest-side shim libraries (/dev/vdb)
- SENS <name> protocol op: canvas-fed named values (built-in sensors /
buttons) served from PiInstance.sensor_state, pushed by the frontend
via the new pi_sensor_state WS message
- DISP <b64> protocol op: guest display commands forwarded to the
frontend as 'display' events (built-in screens)
- RaspberryPi3Bridge: onDisplay / onGpioPwm callbacks + setSensorState
- SimulatorCanvas hands piFamily boards their Pi bridge in
attachBuiltins (was ESP32-only)
add_reader on the proto FIFO armed epoll but never delivered callbacks
under uvloop when the fd number recycled a just-closed socket fd
(nondeterministic per instance): the guest's GPIO lines sat unread in
the pipe and canvas LEDs stayed dark while serial kept flowing. Pump
the FIFO from a worker thread (select + os.read, like _watch_stderr)
and schedule _handle_gpio_line back onto the loop.
registerPiFamilyKind + ProBoardDef.piFamily route an overlay board through
the existing Raspberry Pi bridge path (WebSocket qemu, VFS panel, boot
terminal); register_pi_board_profile lets the overlay add the matching
PI_CONFIGS entry (cpu/image_set/...) at registration time.
Un crash entre ambos (el NameError de board_fqbn) dejaba defaults==rendered
con un sdkconfig stale, y el re-seed de kconfig no disparaba nunca mas en ese
dir persistente — medido: CONSOLE_SECONDARY quedo en usb_serial_jtag pese a
los defaults nuevos. El bloque de velxio_board.cmake pasa a despues.
El bloque de velxio_board.cmake usaba board_fqbn, que vivia en compile() y no
en la firma de _compile_in_dir: parametro nuevo con default None, propagado en
los dos call sites, y guard para el camino sin fqbn (tests directos).
Tres piezas:
- _arduino_board_flags lee <id>.build.board y <id>.build.cdc_on_boot de
boards.txt (cache, mismo patron que _arduino_variant).
- Cada compile escribe velxio_board.cmake (incluido OPTIONAL por el
CMakeLists ANTES de project(), asi los defines llegan a TODOS los
componentes — que Serial sea el CDC se decide dentro del core Arduino):
ARDUINO_<BOARD> siempre que boards.txt lo nombre (los sketches de vendor
hacen #ifdef ARDUINO_XIAO_ESP32S3), y ARDUINO_USB_CDC_ON_BOOT=1 +
ARDUINO_USB_MODE=1 cuando cdc_on_boot=1. USB_MODE va FORZADO a 1 (HWCDC
por USB-Serial-JTAG, la opcion de menu USBMode=hwcdc): el motor modela esa
consola, no la pila OTG/TinyUSB. Ir por archivo y no por env hace el flag
dependencia de configure: CMake reconfigura al cambiar.
- CONFIG_ESP_CONSOLE_SECONDARY_NONE en el sdkconfig: sin ella el ROM/IDF
espejan cada byte de log al USB y el monitor unico del emulador (que
mezcla ambos streams) mostraria el log entero dos veces en builds CDC.
_normalize_options rellena arduinoVariant con _arduino_variant(board_fqbn),
que ya leia boards.txt; la plantilla ya consumia la clave. Sin esto todas las
placas S3 compilaban como esp32s3 generico: sin nombres Dx del XIAO y con
Serial en UART0, cuyo RX por defecto es GPIO44 = D7 del XIAO — un
pinMode(D7) tardio desinstala el driver y silencia los prints del loop
(medido en el banco con el ejemplo del Round Display).
Sin esta linea la variante se queda en el nombre del chip, asi que TODA placa
compila como dev-kit generico y variants/<VARIANTE>/pins_arduino.h nunca entra
en el include path: los nombres propios de la placa (D0..D10 en una XIAO, la
serigrafia de las M5) sencillamente no existen. Justo los sketches que la gente
copia de la wiki del fabricante.
El helper que deriva la variante del FQBN leyendo boards.txt ya estaba; faltaba
la plantilla que la consume. Sigue pendiente rellenar 'arduinoVariant' en
normalized para que surta efecto (ver F7.6 en velxio-prod).
Los dos ejemplos de camara (esp32cam-lcd-preview y esp32cam-webcam-demo) morian
con "esp_camera.h: No such file or directory".
No es que el ejemplo estuviera mal: es una regresion de la migracion. El arbol
viejo /opt/arduino-esp32 (2.x) traia un SDK precompilado con esp32-camera dentro,
en tools/sdk/<target>/include/esp32-camera/, para esp32, c3, s2 y s3. El
componente 3.x que usamos ahora NO lo trae, y su idf_component.yml tampoco
depende de el, asi que el header simplemente dejo de existir. Por eso funcionaba
antes y dejo de funcionar sin que nadie tocara el ejemplo.
Mezclar los headers de la 2.x en este build no es opcion — vienen con su SDK
precompilado y aqui compilamos contra IDF 5. Lo correcto en el mundo 3.x es
pedirle el componente al gestor: la plantilla no traia idf_component.yml, asi que
todo lo gestionado llegaba de rebote por el manifiesto de arduino-esp32, y lo que
el no arrastra hay que pedirlo explicitamente.
_detect_camera_usage() ve el include y _add_managed_components() escribe
main/idf_component.yml con espressif/esp32-camera. Verificado en el contenedor
sobre el mismo directorio que fallaba: el gestor descarga 2.1.7, el configure
termina con esp32-camera en la lista de componentes, y sketch.ino.cpp compila.
Arduino trata igual #include "Lib.h" que #include <Lib.h> para librerias, y
los ejemplos de los propios fabricantes usan la forma con comillas: los de
M5Stack para el Cardputer abren con #include "M5Cardputer.h". Al escanear solo
la forma con angulos, ese sketch no llegaba siquiera al resolutor de librerias y
moria con 'fatal error: M5Cardputer.h: No such file', mientras el mismo sketch
con angulos compilaba sin problema. Cualquiera que pegue un ejemplo oficial se
daba de bruces con esto.
Se pasan ademas los nombres de los ficheros del propio sketch, para que un
#include "Common.h" local siga siendo una cabecera del proyecto y no se
confunda con una libreria.
test/backend/unit/test_espidf_compiler.py cubre ambas formas, el espaciado, las
cabeceras internas de IDF y la cabecera propia del proyecto.
Las dos ramas habian divergido: master llevaba el modo lenguaje ESP-IDF puro
(#139) y v3.2 la ruta de compilacion IDF v5.5 para toda la familia ESP32 mas
los arreglos de venv/toolchain. Ambas tocaban espidf_compiler.py.
Los dos lados son ejes ORTOGONALES y se conservan enteros:
- use_idf5 / arduino_mode (v3.2): que arbol IDF usa el build (5.5 vs 4.4) y
si cabe Arduino-como-componente.
- pure_idf (master): el modo LENGUAJE que elige el usuario; sus ficheros son
las fuentes del componente main con su propio app_main().
Resolucion:
- _build_env acepta los tres. Un build IDF puro fuerza arduino_mode a falso:
la plantilla CMake mete el componente arduino-esp32 en cuanto existe
ARDUINO_ESP32_PATH, asi que dejarlo puesto compilaba el core de Arduino en
un build que no tiene sketch. Lo cazaron los tests de master.
- VELXIO_PURE_SKETCH solo con pure_idf, nunca con arduino_mode a falso a
secas: un target sin core arduino-esp32 sigue entregando un SKETCH al
traductor legacy y no debe tomar la rama del glob puro.
- La identidad del build-dir suma los dos tokens (|idf:N|ard:N y |lang:pure):
ningun par de esas combinaciones puede compartir un build/ configurado.
- La cadena de escritura de fuentes queda pure_idf -> arduino_mode -> legacy.
- sdkconfig: render de v3.2 (con target/use_idf5) mas el filtrado de simbolos
CONFIG_ARDUINO* de master cuando el build es puro.
test/backend/unit/test_espidf_compiler.py: los 7 tests que ya estaban rotos en
v3.2 (AttributeError: idf5_path, fixture sin actualizar desde que se anadio la
seleccion de IDF) vuelven a pasar.
Verificado: backend 293 pasan / 0 fallan (v3.2 traia 7 rotos); frontend 2268
pasan / 0 fallan en los dos shards.
Adds a third entry to the board language selector next to Arduino C++
and MicroPython: ESP-IDF. In this mode the user writes a plain ESP-IDF
project — app_main() entry point, FreeRTOS + driver APIs — and the
backend compiles it through the same ESP-IDF toolchain it already uses
for ESP32 Arduino sketches, just without the arduino-esp32 component.
Backend:
- CompileRequest.language ('espidf') threaded through the sync + async
compile paths and folded into the dedup job key (language='arduino'
and omitted hash identically so old clients keep dedupping).
- espidf_compiler: pure_idf flag. User files are written into main/
as-is (no Arduino.h wrap, no velxio_compat.h, Arduino library
resolution skipped), ARDUINO_ESP32_PATH is dropped from the build env
and VELXIO_PURE_SKETCH raised so the template CMake compiles the
user's own sources via a glob branch. Pure builds get their own
persistent build-dir variant through the eff_hash fold.
- QEMU WiFi compat for IDF-style code: esp_wifi.h/esp_wifi_init
detection sets has_wifi, and literal #define SSID/PASS plus
wifi_config_t designated initializers are normalized to the QEMU AP.
- CONFIG_ARDUINO_* lines are stripped from sdkconfig.defaults in pure
mode (the symbols don't exist without the arduino component).
Frontend:
- LanguageMode gains 'espidf'; BOARD_SUPPORTS_ESPIDF covers the ESP32
family (Xtensa, S3, C3). Toolbar shows the option only for those.
- Switching modes seeds a main.c blink skeleton (app_main + gpio
driver), mirroring the MicroPython main.py flow.
- compileCode sends language='espidf'; run/stop paths are unchanged
(the QEMU worker consumes the same merged flash image).
- New gallery example: esp32-idf-blink (LED + resistor on GPIO 2).
Tests: unit coverage for the build-env switch, IDF wifi normalization,
job-key variance, file-group seeding and the new example; verified
end-to-end in a container from the prod image (pure build produces a
bootable flash image; Arduino-mode build unchanged, same variant hash).
Ports the v5 compiler layer: /opt/esp-idf-v5 + arduino-esp32 3.x
discovery (IDF5_PATH / ARDUINO_ESP32_5_PATH overridable), per-compile IDF
selection routing the family to the v5 tree, esp32c6 target support, and
the docker-runtime hardening (version-matched python venv pinned on PATH,
toolchain ordering by the tree's tools.json, user-libs component REQUIRES
for the IDF components Arduino libraries include directly). The LED_BUILTIN
fallback in the project template becomes a whitelist so non-esp32 variants
(C6 first) never hit the redefinition trap. Includes the option tests.
The newest-env-wins heuristic pinned a 5.x env for a 4.4 tree — each
generation's dependency check only accepts its own requirement set. The
version is read from tools/cmake/version.cmake of the active IDF tree,
falling back to the newest env when unreadable.
Two failure modes seen on a fresh docker deploy where the final image's
system python differs from the one the IDF venv was created with:
- IDF's cmake derives the venv dir name from the SYSTEM python version
and fails with 'python doesn't exist' before configure; and when cmake
is invoked directly (not via idf.py) its PYTHON property falls back to
the bare 'python' from PATH. The Linux branch of _build_env now pins
IDF_PYTHON_ENV_PATH to the newest env under the tools root and puts its
bin first on PATH (mirroring what the Windows branch already did).
- When several toolchain generations share one tools root, glob order
decided which won the PATH race. Toolchain dirs are now ordered by the
versions the IDF tree declares in tools/tools.json.
_emit() wrote to stdout synchronously from QEMU callback context — the
iothread fires _on_uart_tx per UART byte, each becoming ~45 bytes of
JSON. When the parent's pipe reader stalled, the 64 KB pipe filled and
the write blocked INSIDE the QEMU iothread, freezing the entire guest.
Observed as an intermittent (~1 in 10) boot hang: serial output stops
right after the ROM log / rtcinit line — exactly where the accumulated
boot events cross the pipe capacity — and never recovers. Direct worker
harness runs (fast reader) never reproduced it; the browser/WS path did.
Route _emit() through a bounded queue drained by a dedicated writer
thread (opportunistic batching, one write per drain). Under extreme
backpressure events are dropped and counted on stderr — losing telemetry
beats freezing the emulated CPU. The shutdown path posts a sentinel and
joins the writer so crash/system events still flush before os._exit.
Verified on staging: 10/10 UI run/stop cycles + 8/8 direct harness boots
with identical serial latency (~0.95s to first app output).
_refresh_pin_pulls read a fixed 40 registers from get_internals(3), but
the exposed array is per-chip: classic muxgpios[40], ESP32-S3 49
(GPIO0..48). The fixed bound silently missed S3 pulls on GPIO40-48.
Read _GPIO_COUNT entries instead (set from the machine at startup).
Pairs with qemu-lcgamboa 547e989, which models the S3 IO_MUX pull bits
and exposes them via the get_internals override — together they make
INPUT_PULLUP buttons work on the emulated S3 exactly like the classic
ESP32 (verified on staging with GPIO4 and GPIO40).
The S3 has GPIO0..48; the identity pinmap length drives how many pins
picsimlab_wire_gpio connects to the host. Default 40 left GPIO40..48 unwired,
so digitalWrite/Read on those pins never reached the frontend. Build a 49-entry
pinmap when the machine is esp32s3 (mirrors the c3->22 special case). Requires
the GPIO-model widening in libqemu (esp32_gpio bank-1 8->17).
_idf_target had no S3 case, so every S3 FQBN (esp32s3 / XIAO_ESP32S3 /
nano_nora) compiled as plain esp32 (LX6) and could not boot the S3 QEMU
machine. Add _is_esp32s3 (covers all three FQBNs; nano_nora has no 's3'
token) + return 'esp32s3'; add the xtensa-esp32s3-elf toolchain to the
build PATH; place the S3 second-stage bootloader at flash offset 0x0
(like C3) instead of 0x1000 via 'is_c3 or idf_target==esp32s3' at the merge.
Requires the xtensa-esp32s3-elf toolchain in the image
(install.sh esp32,esp32c3,esp32s3) and a libqemu-xtensa with an
esp32s3(-picsimlab) machine.
Extend the spice-driven input path (already live for AVR/ESP32) to RP2040 and
STM32 so digitalRead() of an INPUT pin reflects the actual wiring: a pin tied
to a rail reads that rail, and an INPUT_PULLUP button-to-GND reads idle-HIGH /
pressed-LOW instead of floating or inverted.
RP2040 (rp2040js, frontend-only): the GPIO listener now splits input vs output
mode. Input pins report their pad pull (InputPullUp/Down) via setPinPull and
seed the pull's idle level (rp2040js does not auto-apply the pad pull to the
readable input register); the SPICE solve then overrides via connectDigital-
InputsToMcu when the net is actually sourced. Output pins drive as before.
spiceDrivenInputs = true.
STM32 (backend QEMU): the worker now forwards a new gpio_pull event (from the
libqemu-arm picsimlab_pull_pin callback) so the netlist stamps the matching
weak resistor; Stm32Bridge surfaces it, Stm32BridgeShim opts into
spiceDrivenInputs, and collectPinStates maps PA0/PC13 names to the linear pin
so the pull is read. STM32 outputs stay on the part layer (unchanged).
Event-driven parts with no SPICE model (rotary encoder, keypad) remain
protected by the existing sourcedNets gate in the connector.
Completes the internal-pull emulation for the common case (a button on an
RTC-capable GPIO like 4/15/25/... with INPUT_PULLUP):
- Backend reads the RTC_IO pad RUE/RDE bits via the new
get_internals(QEMU_INTERNAL_RTCIO) and emits gpio_pull for RTC pins, so
pull-up/down on those pads is finally visible (it lives in RTC_IO, not
IO_MUX). IO_MUX path still covers non-RTC pins.
- The digitalRead path is driven by seeding the GPIO input level, not by
SPICE. The part-level INPUT_PULLUP seed (BasicParts) is sent at attach,
before the multi-second QEMU boot finishes, so it is lost and the pin
reads LOW. makePinPullHandler now drives the pin to the pull's idle level
via sendPinEvent when the guest programs the pull (post-boot), so it
sticks. A real button press/release still overrides it.
INPUT_PULLUP / INPUT_PULLDOWN had no effect in simulation: the ESP32's
internal pull resistors live inside QEMU and were invisible to the SPICE
solver, so an input wired to a button-to-GND floated to 0 V and read LOW
even at idle. The canonical active-low button never worked.
Read the pull config straight out of the running guest: the IO_MUX
register (FUN_PU bit 8 / FUN_PD bit 7) is already exposed read-only via
qemu_picsimlab_get_internals(3), so no QEMU rebuild is needed. The worker
scans it on the 100 ms poll thread and emits gpio_pull; the bridge feeds
it to PinManager; the netlist stamps a weak 45k resistor to the rail so
idle inputs read the correct level. 45k matches the real internal pull
and is weak enough that any external driver/pull dominates.
Verified with ngspice: idle ~3.3 V (HIGH), pressed ~0 V (LOW).
- boot_images manifest: bump arm64 rootfs (gpiozero/colorzero baked in,
hostname applied at boot, reworded MOTD)
- RaspberryPi3Bridge: onBooted shell-ready detector + sendAndWaitForPrompt
flow control (resets on disconnect)
- RaspberryPiWorkspace: distinct Booting overlay + piBooted-driven status,
inline SVG icons replacing emoji glyphs
- SerialMonitor: strip CSI/DSR escapes so the dumb console no longer shows
a literal [6n next to the prompt
- VirtualFileSystem: upload auto-starts the Pi and waits for the shell, then
flow-controls each command (no more dropped lines on large files)
- i18n: bootingTitle/bootingNote + reworded offlineNote2 across 9 locales
Move the CYW43439 (Pico W) WiFi emulation out of the open-source tree so it
can ship as a paid feature in a private overlay. OSS keeps a plain Pico W
(no WiFi); the overlay registers the cyw43 protocol + backend network stack
at runtime via generic seams.
Frontend:
- Add simulation/PioPeripheral.ts: a generic "PIO bus peripheral" seam
(feedWord / inDiscardableWriteData / resetFraming / hostWakeLevel /
onHostWake / onSimulationStart). No factory is installed in OSS, so
createPioPeripheral() returns null and a Pico W simulates as a plain Pico.
- RP2040Simulator: keep the fragile PIO-FIFO plumbing (it must re-run after
loadMicroPython swaps the chip) but drive it through PioPeripheral instead
of an inlined cyw43 import (attachCyw43 -> attachPioPeripheral, etc.).
- useSimulatorStore: generic attach/detach + setBoardWifiStatus; drop the
cyw43 bridge map.
- MicroPythonLoader: add registerFirmwareVariant() so an overlay can add the
RPI_PICO_W build; remove the OSS pico-w config + bundled .uf2.
- Delete simulation/cyw43/ (moved to the overlay).
Backend:
- core/hooks.py: add generic register_ws_sim_handler / dispatch_ws_sim_message
and register_gateway_proxy / dispatch_gateway_proxy seams.
- simulation.py: route start_picow / stop_picow / picow_packet_out through the
ws_sim_handler hook (the overlay handles + gates them).
- iot_gateway.py: resolve the Pico W gateway through the gateway_proxy hook.
- Delete services/picow_net/ + picow_net_bridge.py (moved to the overlay).
Tests: move the cyw43/picow suites to the overlay; update RP2040Simulator
mock stubs to attachPioPeripheral.
Two real-world failures hit the Pico W gateway that the headless e2e
(node fetch, tiny request, fast timing) didn't surface:
- A browser sends KILOBYTES of headers (cookies, User-Agent, sec-*).
Forwarded verbatim, the request overran the chip's small recv()
(e.g. recv(1024)); lwIP then RST the connection on close-with-unread-
data, crashing blocking-socket sketches with ECONNRESET. Now we forward
a MINIMAL request (method, path, Host, Connection: close, and
Content-Type/Length for bodies) — nothing a tiny server can choke on.
- After a gateway request completed we dropped the connection immediately,
so a late chip segment (retransmitted FIN / trailing ACK) no longer
matched and fell through to the chip-initiated NAT, which RST it. Add a
short TIME_WAIT: keep the connection briefly and re-ACK late segments so
the chip closes cleanly, never RSTing it.
ESP32 web-server examples are reachable from the browser via
/api/gateway/<client_id>/ (QEMU slirp hostfwd). The Pico W server lives
in the browser-side lwIP, so there was no inbound path: visiting the
chip's IP did nothing.
Add the mirror of tcp_nat.py: tcp_inbound.TcpInbound originates a TCP
connection INTO the chip over the WebSocket bridge (SYN -> SYN+ACK ->
ACK -> request -> response -> FIN), so the backend can fetch a page the
sketch serves on 10.13.37.42:80 and hand it back to the browser.
- bridge.py routes chip TCP segments addressed to a gateway-opened
connection to TcpInbound (before the chip-initiated NAT, which would
RST them); exposes http_into_chip() + ensure_chip_mac() (primes the
chip's gateway ARP).
- iot_gateway.py: same /api/gateway/<client_id>/ route now falls through
to the Pico W bridge when there's no ESP32 instance, builds a raw
HTTP/1.1 request, and parses the chip's response. Same plan gate, same
URL shape — the browser sees no difference between ESP32 and Pico W.
Validated end to end (real RP2040 emulator serving an HTTP page ->
gateway returns it) plus 6 unit tests for the TCP state machine,
response parsing and ARP priming.
Add a working microSD card part backed by a FAT16 image, following the
Wokwi storage model: the project's own workspace files are auto-copied
onto the card (free), and an optional "SD Card" panel uploads extra
files (gated as a paid feature by the velxio.dev overlay; OSS default
allows it).
Frontend (in-browser AVR / RP2040):
- ProtocolParts.ts: rewrite the microsd-card part from a handshake stub
into a real SD-over-SPI device (reply-first Ncr timing, SDSC byte
addressing, single/multi-block read+write, CSD/CID, full CMD set).
- utils/fatImage.ts: dependency-free FAT16 super-floppy builder (8.3 + LFN).
- utils/sdCardFiles.ts: assemble the card image from workspace files plus
uploaded files; base64 helpers.
- components/simulator/SdCardPanel.tsx + ComponentPropertyDialog: upload UI.
- DynamicComponent + useSimulatorStore: build and inject the image on run.
- lib/proSdCardGate.ts: overlay-installable gate for the upload action.
- data/examples-storage-microsd.ts: Arduino Uno + ESP32 gallery examples.
Backend (ESP32 via QEMU):
- services/esp32_sd_slave.py: synchronous SD-over-SPI slave (Python port of
the browser part) with a sparse backing store, idle-state R1 tracking and
real CRC16 on data blocks when the host enables CRC (CMD59) -- both
required by ESP-IDF's sdspi driver.
- esp32_worker.py: route SPI bytes to the slave (returns MISO synchronously)
and feed write-only bulk transfers.
- esp32_lib_manager.py + routes/simulation.py: forward the FAT image
(sd_card.image_b64) from the start config into the worker.
Tested:
- frontend: protocol-parts, fat-image, sd-card-gate and microsd-real-firmware
(real Arduino SD.h on avr8js) -- 86 passing.
- backend: test_esp32_sd_slave (10) covering the ESP-IDF init sequence and
CRC16; validated end to end by running a real SD.h sketch in libqemu-xtensa
(mount, directory listing, read and write-readback).
arduino-esp32's WiFiClientSecure/ssl_client.cpp wraps its ENTIRE body
(start_ssl_client, ssl_init, send_ssl_data, ...) in
#if !defined(MBEDTLS_KEY_EXCHANGE_SOME_PSK_ENABLED) ... #else <body> #endif
ESP-IDF's mbedtls defaults the PSK key-exchange modes OFF, so the object
compiled empty and any sketch using WiFiClientSecure — including
HTTPClient.begin(url), which links the secure client even for http:// —
failed to link with "undefined reference to start_ssl_client". Commenting
out begin() let the optimizer drop the unused client, which is why it
"compiled when commented".
- sdkconfig.defaults.in: enable the PSK key-exchange ciphersuites
(CONFIG_MBEDTLS_PSK_MODES + the four KEY_EXCHANGE_*PSK), matching
arduino-esp32's own sdkconfig.
- espidf_compiler: ESP-IDF only seeds sdkconfig from sdkconfig.defaults when
sdkconfig is ABSENT. Persistent build dirs live in the build volume and
keep a stale sdkconfig across image rebuilds, so the new CONFIG_* would
never reach kconfig. Drop the generated sdkconfig when the rendered
defaults change so it re-seeds on configure.
- test: assert the rendered sdkconfig enables PSK.
Verified end-to-end: the reported WiFi+HTTPClient sketch now compiles to a
1.1 MB binary (was a hard link error before).
list_installed_libraries() now honors VELXIO_FALLBACK_SKETCHBOOK (pro overlay) so
GET /api/libraries/list enumerates the content-addressed cache instead of the
shared global dir — the Installed tab survives the global volume's retirement
(audit finding #3). Unset (OSS self-host) -> default sketchbook, unchanged.
The last paths that read the shared global /root/Arduino/libraries: a compile
with NO manifest (libraries=null — any from-scratch/anon sketch; the manifest is
never auto-derived from #includes) and the incomplete-manifest scan-all retry
(which re-enters compile unscoped). Both fell through to the global dir,
bypassing the cache entirely (a cached lib still failed when global was gone).
Now, when no scope is materialized, point the library search at the cache: the
cache root is itself a valid Arduino libraries dir (each <name@ver-sha> child is
a library), exposed via env (pro overlay sets them):
- arduino-cli: ARDUINO_DIRECTORIES_USER = VELXIO_FALLBACK_SKETCHBOOK (whose
libraries/ -> cache root) when scope_dir is None.
- ESP-IDF: _find_arduino_libraries_dir() prefers VELXIO_FALLBACK_LIBRARIES_DIR.
Unset (OSS self-host) -> legacy global, unchanged.
Also strip a trailing @version from manifest names (_bare_lib_names): norm_name
fused 'ArduinoJson@6.21.5' -> 'arduinojson6215' (cache miss -> global scan-all);
the per-board boards_json manifest is the path that still carries @version.
The compile scope resolved a project OWNER's per-user custom libraries for ANY
project_id a requester supplied, with no visibility check — so a requester who
knew a victim's PRIVATE project_id + custom-lib name could compile a binary
against the victim's private uploaded library. Replace the ungated
get_project_owner hook with resolve_compile_owner(project_id, requester_id),
which returns the owner ONLY when the requester IS the owner OR the project is
shareable (public/unlisted); otherwise None -> the caller falls back to the
requester's OWN store. Fails closed on any error.
Also gate the server-side manifest fallback by the same rule (symmetry): a
private project's declared library NAMES are no longer read into a non-owner's
compile scope. The owner's own compile and shared/embed compiles of public/
unlisted projects keep their backend-authoritative scope unchanged.
Empirically, arduino-cli's --libraries ADDS to the search path — the global
sketchbook is STILL scanned, so the prior commit did NOT isolate reads from the
shared global volume. Point ARDUINO_DIRECTORIES_USER at the scratch sketchbook
instead (its <scratch>/libraries becomes the ONLY user-library dir); cores +
board-manager URLs live in the DATA dir and are untouched.
In-container functest (AVR uno): a cache-only lib compiles via the scope (never
in global); a global-only lib NOT in the manifest is INVISIBLE to the scoped
compile and only recovers via the scan-all retry (manifest_incomplete=True) —
proving the global volume is no longer scanned for a manifest-scoped build.
arduino-cli compiles now read libraries from a per-compile --libraries dir of
symlinks materialized by the pro overlay (owner store -> cache -> legacy), via
the existing materialize_library_scope hook, instead of scanning the shared
mutable global volume. --libraries only overrides the USER library search path,
so cores + board-manager URLs (RP2040 earlephilhower, ATTinyCore) are untouched.
Mirrors the ESP-IDF P2.1e graceful fallback: an incomplete manifest (a needed or
transitive lib not declared) makes the scoped compile miss a header; we retry
ONCE scan-all (no --libraries) and surface manifest_incomplete, so a partial
manifest degrades to legacy behavior instead of hard-failing.
Dedup correctness: the manifest + owner are resolved ONCE (shared
_resolve_compile_scope) and folded into the /compile/start dedup key AND threaded
into the build, so two owners with identical sketch+board but different custom
libs never coalesce to one another's job, and the key never diverges from the
bytes the build uses. Owner folded only when a manifest applies (index-only
compiles keep cross-owner dedup).
New OSS warm_library hook; /api/libraries/install now calls it (with the
requester id for the anon policy) instead of mutating the shared global
libraries volume — so that volume stops growing and can be retired (P2.5).
Falls back to the legacy arduino-cli global install when no overlay is loaded
(OSS self-host parity).
- compile.py: owner_id = project owner ELSE the requester (so an unsaved
compile resolves the libs the user just uploaded, which are their own);
threaded requester_id into _run_compile from both call sites.
- LibraryManagerModal: on a custom .zip upload, auto-add the lib to the active
board's velxio.json + show the Project tab, so the compile resolves it via the
owner per-user path (the upload now lands in the per-user store, not the
shared dir, so it must be declared to be found).
So a scoped compile can resolve the project OWNER's per-user custom libraries
(not the requester's) — a shared/embed/anon compile of someone else's project
still finds that owner's uploaded libs.
- core/hooks.py: new get_project_owner hook; materialize_library_scope gains an
opaque owner_id param (no-op default unchanged).
- espidf_compiler.compile/_attempt: thread owner_id to the materializer.
- compile.py: resolve owner via get_project_owner(project_id), pass to compile.
Additive: the OSS image (no overlay) ignores owner_id; index libs still resolve
from the cache. Foundation for per-user custom-lib storage (P2.2a write side).
A scoped ESP32 compile can now resolve libraries from a per-compile directory
provided by an overlay (the manifest's libs symlinked from a content-addressed
cache, with a legacy-dir fallback) instead of the single shared global volume.
- core/hooks.py: register_materialize_library_scope / materialize_library_scope
(no-op default -> None, so the OSS image keeps its single scan-all dir).
- espidf_compiler: _attempt(allowed) calls the hook, folds the returned content
token into the build-variant eff_hash (a content change gets a clean build
dir), passes libraries_dir to _compile_in_dir (arduino_libs = libraries_dir or
_find_arduino_libraries_dir()), and removes the throwaway dir after. The
graceful scan-all fallback (allowed=None) keeps using the default dir, so the
worst case of any materializer failure is fall-back-to-legacy (no break).
Library manifests are now PER-BOARD (each board carries its own velxio.json),
so two boards in one project can use different (even conflicting) libraries
without clashing — the multi-board extension of the no-clash guarantee.
- board.libraries on BoardInstance + serialisableBoard: rides in boards_json,
so it round-trips, dirty-checks, autosaves and restores natively. This also
removes the load-restore hacks (useLibraryManifestStore + applyProjectManifest
deleted): the manifest is plain board state.
- loadProjectState now restores per-board boardOptions/spiffsFiles/libraries
(it previously dropped them).
- EditorToolbar single + compile-all send the COMPILING board's libraries.
- Backend compile.py prefers the client's per-board request.libraries; the
project-level libraries_json (now the union of all boards) is the fallback.
- buildLoadPayload migrates pre-per-board projects: seed each board with the
project union so they keep compiling scoped.
- Library Manager 'In project' tab edits the ACTIVE board's velxio.json (shows
the board name) and the add field is now an autocomplete (installed libs +
index search) so users pick from a list instead of typing names.
Deletes useLibraryManifestStore.ts + applyProjectManifest.ts.
Adds the get_project_libraries hook: the compile route reads a saved project's
declared library manifest (by project_id) and uses it as the ESP-IDF resolution
scope, preferring it over the client-sent manifest. So a saved project always
compiles against only its own declared libraries — never another user's, or
another project's, stray install in the shared dir — authoritatively from the
server, independent of frontend wiring. Client-sent manifest still used for
unsaved examples; None/empty → legacy scan-all. Overlay fills the hook in
register_pro; OSS default is no-op (None).
Per-variant build dirs fixed the cross-compile staleness, but the FIRST build of
a cold variant can still occasionally hit ESP-IDF nested-build flakiness (cmake /
bootloader / managed-components / sdkconfig). These are infrastructure failures,
never user code, and clear on a retry once the variant dir is warmer. Retry the
attempt once when the failure matches infrastructure markers (not a user-sketch
or missing-library error). Cheap via ccache + ninja incremental.
Replaces the wipe-on-change approach (which left ESP-IDF's nested bootloader /
managed-components build in a broken state under rapid reconfigure -
intermittent 'managed_components_list.temp.cmake: No such file'). Each distinct
configuration (board options x resolved library set, via the variant key the
caller already computes) now gets its OWN persistent project dir with its OWN
build/, never wiped or reconfigured for a different config:
- same config -> same dir -> warm ninja incremental + ccache (fast iterate);
- different config -> different dir -> isolated, consistent, no staleness,
no nested-build breakage;
- the scoped vs scan-all fallback attempts land in different dirs, so the
double-compile no longer corrupts a shared build/.
Variant dirs are LRU-bounded (_MAX_BUILD_VARIANTS per target); the global ccache
warms a fresh/evicted variant in seconds. Cleans up the old single-project
layout on first run. Fixes the cross-compile staleness for legacy AND manifest
compiles, and the fallback regression.
Supersedes the mid-_compile_in_dir build/ wipe (161deb9): wiping build/ AFTER
materializing libs but right before cmake left ESP-IDF's config half-regenerated
during the fallback's scoped->scan-all double-compile, intermittently failing
with 'sdkconfig.h: No such file'.
Instead fold the effective library set (the manifest, else the sketch's
non-core external includes) into the per-attempt build-dir hash, so a changed
lib set — or the scan-all fallback after a scoped attempt — resets the
persistent build/ at _prepare_persistent_project_dir time (before any cmake).
That is the existing, well-tested early-wipe path, so the configure is always
clean. Same lib set across compiles keeps the warm ccache/ninja cache; core-only
sketches share one dir (core headers filtered out of the token) so they never
trigger a spurious wipe.
Fixes both the original cross-compile staleness (intermittent cmake-configure
failures + stale-object false positives) and the fallback regression.
The persistent per-target build/ caches ESP-IDF's cmake configuration, ninja's
incremental graph and ccache-backed objects, all assuming a stable component
set. When consecutive compiles on the same dir have a DIFFERENT resolved
user_libs set (a different project/user, or a different library manifest) that
cache is inconsistent and produces two real failures:
- cmake reconfigure intermittently fails ('cmake configure failed') even
though each manifest compiles fine on a clean dir;
- ninja/ccache reuse a previous compile's objects/headers, letting a
now-absent library slip through as a false-positive success against a lib
the current sketch/manifest no longer includes.
Fingerprint the materialized user_libs/ (sorted relative paths + sizes) and,
when it changes vs the last compile on this dir, wipe build/ to force a clean
configure. ccache (enabled) refills the objects so the rebuild stays cheap.
No-op on the ephemeral path and the first compile. Fixes both symptoms; will be
superseded by the fully ephemeral per-compile workspace (P1).
A manifest-scoped compile that fails because a header isn't in the manifest
(an undeclared/transitive dependency) now retries once with scan-all, so a
project with an incomplete manifest still compiles instead of regressing.
The response reports manifest_incomplete=true and
manifest_suggested_libraries={header: [candidate lib names]} so the manifest
can be auto-completed (P2.4) or the user prompted to add the missing library.
This de-risks turning on manifest sending (P2.3): an incomplete example/project
manifest can never break a build that worked before.
- compile(): _attempt(allowed) helper; retry scan-all on missing-lib failure.
- _missing_library_headers / _suggest_libraries_for_headers helpers.
- CompileResponse.manifest_incomplete + manifest_suggested_libraries.
David Montero Crespo