Commit Graph

33 Commits

Author SHA1 Message Date
David Montero Crespo eb9a3ec92f chore: stop committing package-lock.json (cross-platform breakage)
A lock file pins platform-specific native binaries — Rollup, esbuild, swc.
A lock generated on Windows brings @rollup/rollup-win32-x64-msvc but no
Linux variant; a lock generated on Linux does the inverse. The Docker
build kept blowing up with MODULE_NOT_FOUND on rollup/dist/native.js
whenever the lock came from a contributor's non-Linux machine.

Trade-off: we lose npm's transitive-version pinning. Mitigated by:
- package.json caret ranges keep majors stable
- Docker image is rebuilt + retagged per release, so a deployed image
  has a frozen dep set regardless of the lock
- Production uses a pinned upstream commit via velxio-prod's submodule,
  not lock-driven repro
- Dependabot still flags vulnerable transitives via package.json scans

Changes:
- .gitignore: ignore package-lock.json everywhere
- .dockerignore: same (defense-in-depth — never enter build context)
- Dockerfile.standalone: keep `rm -f package-lock.json` as a safety net
  for `docker build` runs from trees with a local lock
- frontend-tests.yml: `npm ci` → `npm install` (npm ci requires a lock)
- Delete the two committed locks (frontend/ + root). The test/* and
  vscode-extension/* locks are left as-is — internal tooling, separate
  install paths, not in the Docker build.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 01:08:23 -03:00
David Montero Crespo 2a99540e5a fix(docker): drop host package-lock.json before npm install in build
Lock files generated on Windows/macOS pin platform-specific Rollup native
binaries (e.g. @rollup/rollup-win32-x64-msvc) and won't bring in the Linux
ones the Docker image needs. Symptom: `MODULE_NOT_FOUND` on
`rollup/dist/native.js` during `npm run build:docker`. See npm/cli#4828.

Removing the lock inside the build forces a fresh, Linux-native dep
resolution. Side effect: a tiny loss of cross-build version pinning, which
is acceptable here — `npm install` honours the version ranges in
package.json so semver-compatible patches at most slip in.

The proper long-term fix is to regenerate package-lock.json on Linux and
commit that. Until then this rm guards every Docker build.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 01:04:47 -03:00
David Montero Crespo b736aea86a refactor: rename deploy/ → docker/ (container build, not host deploy)
The folder only holds the in-container nginx + entrypoint that the
standalone Dockerfile copies. Calling it "deploy" implied host-level
production glue, which now lives in github.com/velxio/velxio-prod.
"docker/" makes the build-time vs deploy-time split obvious.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 00:05:55 -03:00
David Montero Crespo 531c337d19 fix(install): unblock self-hosting + drop forced wokwi clones
Resolves several install pain points reported by users (#108, #120) and
removes the obligatory upstream-clone step that confused contributors and
slowed down every Docker build.

Install fixes:
- nginx: server_name → catch-all default_server, drop Debian's stock site
  so reverse-proxied users no longer get the "Welcome to nginx" page.
- entrypoint: auto-generate SECRET_KEY at first boot, persisted under
  data/.secret_key. backend/.env is now optional in docker-compose.yml.
- backend: add greenlet>=3.0.0 (SQLAlchemy async dep that was missing on
  some Python builds — caused uvicorn startup failures on WSL).

Wokwi libs come from npm:
- @wokwi/elements 1.9.2, avr8js 0.21.0, rp2040js 1.3.2 are pinned in
  frontend/package.json. Vite aliases removed.
- Dockerfile.standalone no longer clones avr8js / rp2040js / wokwi-elements
  / wokwi-boards. Frontend stage is just COPY + npm install + build:docker.
- Board SVGs vendored under frontend/public/boards/ (10 deduped against
  existing files, 2 truly new). third-party/wokwi-* clones become reference-
  only credits — generate-component-metadata.ts skips gracefully when absent.

Production config split out:
- docker-compose.prod.yml, deploy/nginx.prod.conf, nginx-host-velxio*.conf,
  update-third-party.bat removed. Production deployment lives in its own
  repo: https://github.com/velxio/velxio-prod (host nginx + HTTPS + backups
  + pinned upstream commit).

Verified locally: 1161 frontend tests pass, build:docker completes clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 00:04:11 -03:00
David Montero Crespo 9cd5061732 refactor: rename wokwi-libs/ → third-party/
The directory grew well beyond Wokwi-only contents: it now hosts
lcgamboa's QEMU fork (qemu-lcgamboa), Espressif's esp32-camera, the
ngspice WASM build, fritzing-parts, picowi, an alternative QEMU
(qemu-esp32), the 100_Days_100_IoT_Projects examples repo, and
Wokwi's own avr8js/rp2040js/wokwi-elements/wokwi-features/wokwi-boards.
"wokwi-libs" was misleading — half the contents have nothing to do
with Wokwi. "third-party/" is the standard convention for vendored
external dependencies.

Mechanical changes:

  Path rename:
    wokwi-libs/ → third-party/
    update-wokwi-libs.bat → update-third-party.bat
    docs/WOKWI_LIBS.md → docs/THIRD_PARTY.md

  Submodule reconfiguration:
    .gitmodules — 4 path= and section names updated
    .git/modules/wokwi-libs/ → .git/modules/third-party/
    each submodule's .git file rewired to ../../.git/modules/third-party/<name>

  Reference updates (~80 files): vite.config.ts aliases, Dockerfile
    COPY paths, GH Actions workflow steps, build_qemu_*.sh, all
    docs/* and test/*/autosearch/* entries that mention the path,
    package-lock.json file: dependencies, .gitignore patterns,
    sitemap.xml + index.html SEO blurbs, scripts/generate-component-*,
    .dockerignore, .idea/vcs.xml. Bulk replaced both `wokwi-libs/`
    (path) and bare `wokwi-libs` (textual mentions in docs/comments).

Verified:
  - npx tsc -b --noEmit produces no new errors related to these paths
  - vite.config.ts aliases now point at ../third-party/avr8js etc.
  - All 4 git submodules (avr8js, rp2040js, wokwi-elements,
    wokwi-features) are linked under third-party/ with their
    worktrees re-populated and config files referencing the new path
  - `grep -r wokwi-libs` returns zero hits outside node_modules,
    .vite, frontend/dist, third-party/ (upstream submodule contents),
    *.pyc caches, and *.dll.pre-camera rollback binaries

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 00:58:57 -03:00
David Montero Crespo 6a88375bc0 feat: persist multi-board projects + add auto-save
The project save/load pipeline only persisted a single `board_type`, so
multi-board workspaces silently lost every board except the active one
on save, and wires referencing the dropped boards' IDs orphaned to the
canvas corner on reload. An audit of the production backup found 74/306
projects (24%) with at least one orphaned wire and 174/301 non-trivial
projects whose code was still the default Blink template — strong signal
that users save once and never re-save.

Backend
- Add `boards_json` column on `projects` with idempotent ALTER TABLE in
  the lifespan migration list.
- New `FileGroup` schema + `file_groups` array on
  ProjectCreate/Update/Response. Legacy `files`/`code` kept for back-compat.
- `project_files.py` now uses `{pid}/{groupId}/{filename}` subdirs via
  `read_groups`/`write_groups`. Legacy flat layouts are auto-promoted on
  read; legacy single-list `files` only updates the active group, leaving
  other boards' files intact.
- `_persist_files_from_body` honors file_groups → files → code priority.

Frontend
- `useSimulatorStore.addBoard` accepts an optional `explicitId` so
  saved board IDs can be restored verbatim (wires reference IDs literally).
- New `loadProjectState({boards, fileGroups, components, wires,
  activeBoardId})` action: tears down current boards, recreates from the
  payload, restores file groups atomically, recalculates wire positions
  on the next frame, and refreshes the Interconnect.
- `useEditorStore.replaceFileGroups` for atomic multi-group restore.
- `SaveProjectModal` and `ProjectByIdPage`/`ProjectPage` now go through
  `buildSavePayload` / `buildLoadPayload` (handles pre-backfill projects
  by synthesising a default board from `board_type`).

Auto-save (#useAutoSaveProject hook)
- 2.5s debounced silent PUT triggered ONLY when an authenticated user
  has a `currentProject` with a UUID. State hash detects real changes
  vs. UI-only churn; baseline is reset on project load so the just-loaded
  state isn't immediately re-saved.
- `beforeunload` flush via `fetch keepalive: true` (supports PUT +
  credentials, survives unload).
- Compact status indicator in `AppHeader` (idle/dirty/saving/saved/error).

Backfill script (one-off, idempotent)
- `backend/scripts/backfill_boards_2026_05.py` populates `boards_json`
  for legacy projects. Heuristic per project, based on which board IDs
  the wires reference:
    Case A — wires only ref 'arduino-uno' but board_type ≠ uno:
             rename id→board_type and rewrite wire endpoints.
    Case B — single-board normal: keep verbatim.
    Case C — multi-board: recreate one board per distinct ref, infer
             kind by stripping trailing -N suffix.
  Also moves any flat files into the active board's group subdir.
  Stdlib-only, runs from host or `docker exec`.

Docker
- `Dockerfile.standalone` now copies `backend/scripts/` into the image
  so the backfill is callable via `docker exec velxio-app python
  /app/scripts/backfill_boards_2026_05.py --apply`.

Verified locally on the restored production backup (363 projects):
33 Case A, 316 Case B, 14 Case C, 135 wire endpoints renamed, 0 orphans.
Re-running the script after apply skips all 363 (idempotent).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 13:43:33 -03:00
David Montero Crespo 2927a3a2d7 feat: add scripts directory to Dockerfile and mark subproject commits as dirty 2026-04-14 17:38:03 -03:00
David Montero Crespo a3db014d2d feat: multi-arch Docker (amd64+arm64) and fix LED ground check
Docker multi-arch:
- Dockerfile downloads arch-specific QEMU .so via TARGETARCH
- docker-publish.yml adds setup-qemu-action and platforms: linux/amd64,linux/arm64
- qemu-lcgamboa submodule updated (matrix build for both architectures)

LED fix:
- LEDs now require cathode wired to GND (or LOW GPIO) to light up
- Previously LEDs turned on with anode HIGH regardless of cathode connection
- Updated tests to verify anode+cathode behavior
2026-04-07 03:58:52 -03:00
David Montero Crespo b73f28c174 feat: add pre-built ESP-IDF toolchain image workflow 2026-04-04 18:03:43 -03:00
David Montero Crespo 0a724b7566 feat: pre-built QEMU binaries from GitHub Release + WiFi SSID normalization
- Dockerfile: download pre-built .so + ROM from velxio public release
  instead of building from private qemu-lcgamboa source
- espidf_compiler: normalize any WiFi SSID → "Velxio-GUEST" for QEMU
  compatibility (channel 6, open auth)
- docker-compose.yml: unified dev/prod using Dockerfile.standalone
- .dockerignore: exclude qemu-lcgamboa source from Docker context
- .gitignore: ignore prebuilt/ binaries, keep .gitkeep

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-04 14:27:52 -03:00
David Montero Crespo f495a2ce3a feat: implement ESP32 QEMU backend manager and frontend simulation interface 2026-04-01 22:41:52 -03:00
David Montero ba8b56c647 fix: install ESP-IDF Python requirements in final Docker stage
ESP-IDF 4.4.x requires specific package versions (reedsolo<=1.5.4,
bitstring<4, pyparsing<2.4.0, kconfiglib==13.7.1, etc.) that conflict
with newer defaults. Installing requirements.txt after copying ESP-IDF
ensures the correct versions are used and cmake configure succeeds.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 07:45:47 +02:00
David Montero 372e4aea38 fix: add git and packaging to runtime image for ESP-IDF cmake
- git is required by ESP-IDF cmake to resolve submodules and version info
- packaging Python package is required by ESP-IDF Python dependency checks
  during cmake configure step

Without these, ESP32 compilation fails at cmake configure with:
  "Some Python dependencies must be installed"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 07:39:46 +02:00
David Montero 447ab872f5 fix: add cmake, ninja-build and libusb to runtime image
- cmake and ninja are required by ESP-IDF compilation pipeline but were
  missing from the final stage, causing all ESP32 compilations to fail
- libusb-1.0-0 fixes the openocd-esp32 shared library warning on startup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 07:28:01 +02:00
David Montero Crespo 54f8f2782b feat: add ESP32 WiFi/BLE emulation with ESP-IDF compilation pipeline
Replace arduino-cli with ESP-IDF 4.4.7 for ESP32 compilation — Arduino-compiled
firmware crashes in QEMU (9-28 reboots) while ESP-IDF boots cleanly (0 reboots).
The new espidf_compiler translates Arduino WiFi/WebServer sketches to native
ESP-IDF C code, compiles with cmake+ninja, and merges into 4MB flash images.

Key changes:
- ESP-IDF compiler: translates WiFi.begin/WebServer to esp_wifi/esp_http_server
- ESP-IDF project template with QEMU-optimized sdkconfig (DIO, 40MHz, no WDT)
- WiFi status parser for ESP-IDF serial logs (wifi_status, ble_status events)
- IoT Gateway HTTP reverse proxy for ESP32 web servers
- WiFi/BLE auto-detection from sketch content + visual status icons
- Static IP 192.168.4.15 matching slirp DHCP first-client range
- Docker: new espidf-builder stage with ESP-IDF 4.4.7 toolchain
- 157 tests covering WiFi/BLE for both ESP32 (Xtensa) and ESP32-C3 (RISC-V)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 20:53:56 -03:00
David Montero Crespo 8f0c23c857 feat: optimize Dockerfile by deferring arduino-cli core installations to entrypoint.sh for a smaller image size 2026-03-24 18:20:38 -03:00
David Montero Crespo f4e4f863a9 feat: simplify Dockerfile by removing redundant cp command; mark subproject commits as dirty 2026-03-24 17:59:04 -03:00
David Montero Crespo 8ed0438abf feat: update docker-compose to include arduino-libs volume; mark subproject commits as dirty 2026-03-24 17:30:56 -03:00
David Montero Crespo 528287db95 feat: add libfdt1 dependency in Dockerfile; mark subproject commits as dirty 2026-03-24 17:28:56 -03:00
David Montero Crespo 5260261201 feat: Add wokwi-boards library to Dockerfile and mark subprojects as dirty 2026-03-23 11:34:23 -03:00
David Montero Crespo 7053b6f2c8 feat: update ESP32-C3 simulator and add emulation tests
- Updated components-metadata.json with new generated timestamp.
- Refactored Esp32C3Simulator.ts to remove unnecessary debug variables and logging, and added support for additional ROM functions.
- Modified useSimulatorStore.ts to clarify bridge usage for ESP32 boards.
- Updated submodules for QEMU and other libraries to indicate dirty state.
- Added test_esp32c3_emulation.py for end-to-end testing of ESP32-C3 emulation, including compilation, flash image merging, and GPIO event checking.
2026-03-18 23:30:45 -03:00
David Montero Crespo a99a9d512f feat: enhance ESP32 emulation with QEMU integration and update documentation 2026-03-14 14:35:59 -03:00
David Montero Crespo 03f2d7f22e fix: persist SQLite database via host bind mount
Mount ./data:/app/data in both compose files and set DATABASE_URL to
the absolute path /app/data/velxio.db so the database survives container
restarts and updates. Create /app/data in Dockerfile.standalone.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-06 20:32:36 -03:00
David Montero Crespo 22de9173ba fix: update Dockerfile to clone wokwi-libs directly and streamline build process 2026-03-06 13:53:26 -03:00
David Montero Crespo 9c376e70d4 fix: update rp2040js build process to conditionally run based on package.json presence 2026-03-06 13:31:13 -03:00
David Montero Crespo 862d90c2b7 fix: streamline Docker build process and update build scripts 2026-03-06 13:17:14 -03:00
David Montero Crespo 2a8d126b2b fix: update build commands to run only if scripts are present 2026-03-06 13:01:13 -03:00
David Montero Crespo 16271c4bd1 fix: update frontend builder image to use full node:20 version 2026-03-06 12:09:55 -03:00
David Montero Crespo 0d5d440a56 fix: enhance Docker build process and improve file explorer resizing functionality 2026-03-06 11:20:47 -03:00
David Montero Crespo 34dd56b789 fix: retry full build install in docker 2026-03-06 11:05:55 -03:00
David Montero Crespo d0474a5e59 fix: explicit package installs for metadata generation 2026-03-06 11:02:22 -03:00
David Montero Crespo 8f0c431f8d fix: checkout submodules and use robust docker build stage 2026-03-06 10:59:22 -03:00
David Montero Crespo a5c6987aca feat: implement user authentication and project management features
- Add LoginPage and RegisterPage for user authentication.
- Create UserProfilePage to display user projects.
- Implement ProjectPage for viewing and editing individual projects.
- Introduce authService for handling user login, registration, and session management.
- Add projectService for managing project data retrieval and manipulation.
- Enhance EditorPage with file management capabilities and save prompts.
- Introduce Zustand stores for managing authentication, editor state, and project state.
- Add reserved usernames utility to prevent certain usernames during registration.
- Update compilation service to handle multiple files for Arduino sketches.
2026-03-06 10:14:50 -03:00