test(load-test): task sesi interaktif + parser konten rekursif + e2e smoke check
- content_parser.py: scan rekursif subfolder (dasar/arduino/circuit) + inject 50 token - locustfile.py: task interactive_session_python (create→poll→input→verify→delete) - e2e_session_check.py: smoke-test 13 skenario via proxy Flask - README: skenario load test 50 pengguna 2 device chore(elemes): update velxio submodule ke elemes-v3.2 (2b17470)
This commit is contained in:
a2nr
parent
ec66d794a8
commit
575787af59
|
|
@ -14,26 +14,27 @@ source ./env/bin/activate
|
|||
# 1. Install dependency
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 2. Generate test data dari content/
|
||||
python content_parser.py
|
||||
# 2. Generate test data dari content/ (konten ada di subfolder dasar/, arduino/, circuit/)
|
||||
python content_parser.py --content-dir ../../content --tokens-file ../../tokens_siswa.csv --num-tokens 50
|
||||
|
||||
# 3. Jalankan Locust (opsional: set VELXIO_HOST jika Velxio bukan di localhost:8001)
|
||||
export VELXIO_HOST=http://localhost:8001
|
||||
locust -f locustfile.py
|
||||
```
|
||||
|
||||
Buka **http://localhost:8089**, masukkan URL backend Elemes (misalnya `http://localhost:5000`), lalu mulai test.
|
||||
Buka **http://localhost:8089**, masukkan URL backend Elemes (misalnya `http://localhost:3000`), lalu mulai test.
|
||||
|
||||
## File
|
||||
|
||||
| File | Fungsi |
|
||||
|------|--------|
|
||||
| `content_parser.py` | Parse `content/*.md` → `test_data.json` + inject token test ke CSV |
|
||||
| `locustfile.py` | Locust script (7 task weighted) yang baca `test_data.json` |
|
||||
| `content_parser.py` | Parse `content/**/*.md` (rekursif subfolder) → `test_data.json` + inject token `LOCUST_TEST_*` ke CSV |
|
||||
| `locustfile.py` | Locust script (task weighted) yang baca `test_data.json` |
|
||||
| `test_data.json` | Auto-generated, **jangan di-commit** |
|
||||
| `requirements.txt` | Dependency (`locust`) |
|
||||
| `e2e_session_check.py` | Smoke-test alur interaktif (PTY session) via Flask proxy — jalan di container |
|
||||
|
||||
## Test Scenarios (8 Tasks)
|
||||
## Test Scenarios (9 Tasks)
|
||||
|
||||
| # | Task | Weight | Target | Deskripsi |
|
||||
|---|------|--------|--------|----------|
|
||||
|
|
@ -41,17 +42,63 @@ Buka **http://localhost:8089**, masukkan URL backend Elemes (misalnya `http://lo
|
|||
| 2 | View Detail | 5 | Elemes | GET `/lesson/{slug}.json`, validasi field per tipe |
|
||||
| 3 | Compile C | 4 | Elemes | POST `/compile`, validasi output vs expected |
|
||||
| 4 | Compile Python | 3 | Elemes | POST `/compile`, validasi output vs expected |
|
||||
| 5 | Verify Arduino | 2 | Elemes | GET lesson Arduino, validasi JSON structure |
|
||||
| 6 | Complete Flow | 2 | Both | fetch → compile → track-progress |
|
||||
| 7 | Progress Report | 1 | Elemes | Login guru → GET `/progress-report.json` |
|
||||
| 8 | **Compile Arduino** | **3** | **Velxio** | POST `/api/compile`, validasi hex_content |
|
||||
| 5 | **Interactive Session Python** | **4** | **Elemes** | POST `/compile/sessions` → poll prompt → POST `/input` → verifikasi output → DELETE |
|
||||
| 6 | Verify Arduino | 2 | Elemes | GET lesson Arduino, validasi JSON structure |
|
||||
| 7 | Complete Flow | 2 | Both | fetch → compile → track-progress |
|
||||
| 8 | Progress Report | 1 | Elemes | Login guru → GET `/progress-report.json` |
|
||||
| 9 | **Compile Arduino** | **3** | **Velxio** | POST `/api/compile`, validasi hex_content |
|
||||
|
||||
> **Task 5 (interactive session)** menguji alur baru playground: Run → prompt muncul → ketik jawaban → Enter → output. Skrip memakai token (login) agar lolos rate-limit anonymous dan benar-benar mengukur kapasitas compiler worker (max 50 sesi, 2 compile bersamaan, queue timeout 20 s).
|
||||
|
||||
## Load Test 50 Pengguna (2 Device)
|
||||
|
||||
Host hanya 4-core/3.5 GiB — 50 kompilasi berat bersamaan tidak realistis, tapi 50 sesi yang **menunggu input** aman (proses idle tidak makan CPU). Batas worker: `INTERACTIVE_MAX_SESSIONS=50`, `INTERACTIVE_MAX_COMPILES=2`, `INTERACTIVE_QUEUE_TIMEOUT_SECONDS=20`.
|
||||
|
||||
### Device A — Locust Master (jalankan UI di sini)
|
||||
|
||||
```bash
|
||||
cd elemes/load-test
|
||||
source ./env/bin/activate
|
||||
locust -f locustfile.py --master --web-port 8089
|
||||
```
|
||||
|
||||
### Device B (atau beberapa device) — Locust Worker
|
||||
|
||||
```bash
|
||||
cd elemes/load-test
|
||||
source ./env/bin/activate
|
||||
# Ganti <IP_MASTER> dengan IP device A (di jaringan/Tailscale yang sama)
|
||||
locust -f locustfile.py --worker --master-host <IP_MASTER>
|
||||
```
|
||||
|
||||
### Mulai Test
|
||||
|
||||
1. Buka `http://<IP_MASTER>:8089` di browser device A.
|
||||
2. **Host**: URL app yang diuji, misal `https://sinau-c-dev.manakin-gentoo.ts.net` (Tailscale Funnel) atau `http://localhost:3000`.
|
||||
3. **Number of users**: 50 · **Spawn rate**: 5–10/s (ramp-up ~5–10 detik agar hampir bersamaan) · **Duration**: 5–10 menit.
|
||||
4. Amati di tab Charts: RPS per endpoint, response time `/compile/sessions` & `/input`, dan failure rate.
|
||||
5. Skenario `/compile/sessions [Python interactive]` wajib ≥95% sukses; failure `429` pada create menandakan anon rate limit (bukan kapasitas) — gunakan token.
|
||||
|
||||
> Catatan: task compile biasa (`/compile`) tetap kena anon rate limit `1 per 2 menit` bila token tidak ikut — login di `on_start` menyimpan cookie, jadi user ber-token exempt.
|
||||
|
||||
## Re-generate Setelah Tambah Lesson Baru
|
||||
|
||||
Setiap kali ada lesson baru di `content/`, cukup jalankan ulang:
|
||||
|
||||
```bash
|
||||
python content_parser.py
|
||||
python content_parser.py --content-dir ../../content --tokens-file ../../tokens_siswa.csv
|
||||
```
|
||||
|
||||
`test_data.json` akan di-update otomatis dan Locust langsung test lesson baru.
|
||||
|
||||
## E2E Smoke Test (container)
|
||||
|
||||
Verifikasi cepat alur interaktif tanpa browser — jalan di dalam container backend:
|
||||
|
||||
```bash
|
||||
podman cp elemes/load-test/e2e_session_check.py lms-dev_elemes_1:/tmp/
|
||||
podman exec -e E2E_TOKEN=<token_siswa> lms-dev_elemes_1 python3 /tmp/e2e_session_check.py
|
||||
# Contoh token: LOCUST_TEST_xxxx (ambil dari tokens_siswa.csv)
|
||||
```
|
||||
|
||||
Tanpa `E2E_TOKEN` skrip tetap jalan tapi hanya 2 create session (anon rate limit 1/2 menit).
|
||||
|
|
|
|||
|
|
@ -62,6 +62,15 @@ def detect_lesson_type(content: str) -> str:
|
|||
return 'c'
|
||||
|
||||
|
||||
def find_lesson_file(content_dir: str, slug: str) -> str | None:
|
||||
"""Cari file .md dengan slug (basename) di content_dir secara rekursif."""
|
||||
target = f'{slug}.md'
|
||||
for root, _dirs, files in os.walk(content_dir):
|
||||
if target in files:
|
||||
return os.path.join(root, target)
|
||||
return None
|
||||
|
||||
|
||||
def parse_lesson(filepath: str) -> dict:
|
||||
"""Parse a single lesson markdown file and extract test data."""
|
||||
with open(filepath, 'r', encoding='utf-8') as f:
|
||||
|
|
@ -123,6 +132,22 @@ def parse_lesson(filepath: str) -> dict:
|
|||
return data
|
||||
|
||||
|
||||
def scan_all_lessons(content_dir: str) -> list[str]:
|
||||
"""Scan recursively for lesson slugs (basename without .md).
|
||||
|
||||
Konten tersimpan di subfolder (dasar/, arduino/, circuit/), sedangkan
|
||||
API lesson memakai slug tanpa folder (find_lesson_file mencari di semua
|
||||
folder). Sub-home.md bukan lesson, di-skip.
|
||||
"""
|
||||
slugs = []
|
||||
for root, _dirs, files in os.walk(content_dir):
|
||||
for f in sorted(files):
|
||||
if not f.endswith('.md') or f in ('home.md', 'sub-home.md'):
|
||||
continue
|
||||
slugs.append(f[:-3])
|
||||
return sorted(set(slugs))
|
||||
|
||||
|
||||
def get_ordered_slugs(content_dir: str) -> list[str]:
|
||||
"""Get lesson slugs in order from home.md's Available_Lessons section."""
|
||||
home_path = os.path.join(content_dir, 'home.md')
|
||||
|
|
@ -239,21 +264,23 @@ def main():
|
|||
|
||||
# 1. Get ordered lesson slugs
|
||||
ordered_slugs = get_ordered_slugs(content_dir)
|
||||
all_slugs = scan_all_lessons(content_dir)
|
||||
if not ordered_slugs:
|
||||
# Fallback: scan directory
|
||||
ordered_slugs = [
|
||||
f.replace('.md', '')
|
||||
for f in sorted(os.listdir(content_dir))
|
||||
if f.endswith('.md') and f not in ('home.md', 'sub-home.md')
|
||||
]
|
||||
# Fallback: recursive scan (konten di subfolder)
|
||||
ordered_slugs = all_slugs
|
||||
else:
|
||||
# Gabungkan: slug dari home.md (urutan) + slug hasil scan yang belum ada
|
||||
for s in all_slugs:
|
||||
if s not in ordered_slugs:
|
||||
ordered_slugs.append(s)
|
||||
|
||||
print(f" 📚 Found {len(ordered_slugs)} lessons:")
|
||||
|
||||
# 2. Parse each lesson
|
||||
lessons = []
|
||||
for slug in ordered_slugs:
|
||||
filepath = os.path.join(content_dir, f'{slug}.md')
|
||||
if not os.path.exists(filepath):
|
||||
filepath = find_lesson_file(content_dir, slug)
|
||||
if not filepath:
|
||||
print(f" ⚠ {slug}.md not found, skipping")
|
||||
continue
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,139 @@
|
|||
#!/usr/bin/env python3
|
||||
"""E2E check alur interaktif playground via proxy Flask (di dalam container)."""
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
|
||||
BASE = "http://127.0.0.1:5000"
|
||||
TOKEN = __import__("os").environ.get("E2E_TOKEN", "").strip()
|
||||
|
||||
|
||||
def _with_token(payload):
|
||||
if TOKEN:
|
||||
payload = dict(payload)
|
||||
payload["token"] = TOKEN
|
||||
return payload
|
||||
|
||||
|
||||
def call(method, path, payload=None, params=None):
|
||||
url = BASE + path
|
||||
if params:
|
||||
url += "?" + "&".join(f"{k}={v}" for k, v in params.items())
|
||||
data = json.dumps(payload).encode() if payload is not None else None
|
||||
req = urllib.request.Request(url, data=data, method=method)
|
||||
if data:
|
||||
req.add_header("Content-Type", "application/json")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=15) as r:
|
||||
return r.status, json.loads(r.read().decode())
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode()
|
||||
try:
|
||||
return e.code, json.loads(body)
|
||||
except Exception:
|
||||
return e.code, {"raw": body[:200]}
|
||||
|
||||
|
||||
def wait_for(sid, predicate, timeout=15, poll=0.3, cursor=0):
|
||||
"""Poll output; return (status_dict, final_cursor)."""
|
||||
last = None
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < timeout:
|
||||
st, body = call("GET", f"/compile/sessions/{sid}", params={"cursor": cursor})
|
||||
if st != 200:
|
||||
last = ("ERR", st, body)
|
||||
time.sleep(poll)
|
||||
continue
|
||||
cursor = body.get("cursor", cursor)
|
||||
last = body
|
||||
if predicate(body):
|
||||
return body, cursor
|
||||
time.sleep(poll)
|
||||
return last, cursor
|
||||
|
||||
|
||||
results = []
|
||||
|
||||
|
||||
def check(name, cond, detail=""):
|
||||
results.append((name, bool(cond), detail))
|
||||
print(f"{'PASS' if cond else 'FAIL'} {name}" + (f" {detail}" if detail and not cond else ""))
|
||||
|
||||
|
||||
# ── 1. Python interaktif: Run → prompt → input → output ──────────────
|
||||
st, body = call("POST", "/compile/sessions", _with_token({
|
||||
"language": "python",
|
||||
"files": [{"name": "main.py", "content": 'nama = input("Siapa nama kamu? ")\nprint(f"Halo, {nama}!")\n'}],
|
||||
"active_file": "main.py",
|
||||
}))
|
||||
check("py: create session", st in (200, 202) and body.get("session_id"), f"st={st} body={body}")
|
||||
sid = body.get("session_id", "")
|
||||
|
||||
if sid:
|
||||
b, cur = wait_for(sid, lambda x: "Siapa nama kamu?" in x.get("output", ""))
|
||||
check("py: prompt muncul (tanpa stdin, tetap running)", b.get("status") == "running" and "Siapa nama kamu?" in b.get("output", ""), f"{b.get('status')} out={b.get('output')!r}")
|
||||
|
||||
st, b2 = call("POST", f"/compile/sessions/{sid}/input", {"text": "anggoro"})
|
||||
check("py: input kirim OK", st == 200, f"st={st}")
|
||||
|
||||
b3, _ = wait_for(sid, lambda x: x.get("status") in ("exited", "error"), cursor=cur)
|
||||
out = b3.get("output", "")
|
||||
check("py: output Halo, anggoro!", "Halo, anggoro!" in out and b3.get("exit_code") == 0, f"status={b3.get('status')} exit={b3.get('exit_code')} out={out!r}")
|
||||
check("py: tidak ada EOFError", "EOFError" not in out, out[-200:])
|
||||
|
||||
st, _ = call("DELETE", f"/compile/sessions/{sid}")
|
||||
check("py: delete idempotent", st in (200, 404), f"st={st}")
|
||||
|
||||
# ── 2. C interaktif: scanf prompt → input → output ───────────────────
|
||||
C_PROMPT = '#include <stdio.h>\nint main() {\n char nama[64];\n printf("Nama: ");\n fflush(stdout);\n scanf("%63s", nama);\n printf("Halo, %s!\\n", nama);\n return 0;\n}\n'
|
||||
st, body = call("POST", "/compile/sessions", _with_token({
|
||||
"language": "c",
|
||||
"files": [{"name": "main.c", "content": C_PROMPT}],
|
||||
"active_file": "main.c",
|
||||
}))
|
||||
check("c: create session", st in (200, 202) and body.get("session_id"), f"st={st} body={body}")
|
||||
sid = body.get("session_id", "")
|
||||
if sid:
|
||||
b, cur = wait_for(sid, lambda x: "Nama:" in x.get("output", ""), timeout=20)
|
||||
check("c: prompt Nama: muncul", b.get("status") == "running" and "Nama:" in b.get("output", ""), f"{b.get('status')} out={b.get('output')!r}")
|
||||
st, _ = call("POST", f"/compile/sessions/{sid}/input", {"text": "Budi"})
|
||||
b3, _ = wait_for(sid, lambda x: x.get("status") in ("exited", "error"), cursor=cur, timeout=20)
|
||||
check("c: output Halo, Budi!", "Halo, Budi!" in b3.get("output", "") and b3.get("exit_code") == 0, f"status={b3.get('status')} out={b3.get('output')!r}")
|
||||
call("DELETE", f"/compile/sessions/{sid}")
|
||||
|
||||
# ── 3. C multi-file dengan foo.h (include header) ────────────────────
|
||||
FILES = [
|
||||
{"name": "main.c", "content": '#include <stdio.h>\n#include "foo.h"\nint main() {\n printf("Hasil: %d\\n", tambah(7, 8));\n return 0;\n}\n'},
|
||||
{"name": "foo.h", "content": "#ifndef FOO_H\n#define FOO_H\nstatic inline int tambah(int a, int b) { return a + b; }\n#endif\n"},
|
||||
]
|
||||
st, body = call("POST", "/compile/sessions", _with_token({"language": "c", "files": FILES, "active_file": "main.c"}))
|
||||
check("c-multi: create session", st in (200, 202) and body.get("session_id"), f"st={st} body={body}")
|
||||
sid = body.get("session_id", "")
|
||||
if sid:
|
||||
b, _ = wait_for(sid, lambda x: x.get("status") in ("exited", "error"), timeout=25)
|
||||
out = b.get("output", "")
|
||||
check("c-multi: foo.h ikut dikompilasi (Hasil: 15)", "Hasil: 15" in out and b.get("exit_code") == 0, f"status={b.get('status')} out={out!r}")
|
||||
|
||||
# ── 4. Python tanpa stdin: tetap aktif, bukan EOFError ───────────────
|
||||
st, body = call("POST", "/compile/sessions", _with_token({
|
||||
"language": "python",
|
||||
"files": [{"name": "main.py", "content": 'x = input("Angka: ")\nprint(x)\n'}],
|
||||
}))
|
||||
sid = body.get("session_id", "")
|
||||
if sid:
|
||||
b, _ = wait_for(sid, lambda x: "Angka:" in x.get("output", ""), timeout=15)
|
||||
check("py-nostdin: prompt tetap aktif (bukan EOF)", b.get("status") == "running", f"status={b.get('status')}")
|
||||
call("DELETE", f"/compile/sessions/{sid}")
|
||||
|
||||
# ── 5. Error handling: sesi invalid → 404 ────────────────────────────
|
||||
st, body = call("GET", "/compile/sessions/tidak-ada")
|
||||
check("404 session unknown", st == 404, f"st={st}")
|
||||
|
||||
fails = [r for r in results if not r[1]]
|
||||
print("\n===== HASIL =====")
|
||||
print(f"{len(results) - len(fails)}/{len(results)} PASS")
|
||||
if fails:
|
||||
print("FAIL:", [(r[0], r[2]) for r in fails])
|
||||
sys.exit(1)
|
||||
|
|
@ -248,6 +248,113 @@ class ElemesStudent(HttpUser):
|
|||
f"not in output '{output[:50]}'"
|
||||
)
|
||||
|
||||
# ── Task 4b: Interactive Session Python (weight=4) ────────────────
|
||||
# Alur baru playground: Run → prompt → ketik jawaban → Enter → output.
|
||||
|
||||
@task(4)
|
||||
def interactive_session_python(self):
|
||||
"""Buat sesi interaktif Python, kirim input, verifikasi output.
|
||||
|
||||
Memakai token (login) agar lolos rate-limit anonymous dan mengukur
|
||||
kapasitas compiler worker (max 50 sesi, 2 compile bersamaan).
|
||||
"""
|
||||
code = (
|
||||
'nama = input("Siapa nama kamu? ")\n'
|
||||
'print(f"Halo, {nama}! Selamat belajar pemrograman.")\n'
|
||||
)
|
||||
payload = {
|
||||
'language': 'python',
|
||||
'files': [{'name': 'main.py', 'content': code}],
|
||||
'active_file': 'main.py',
|
||||
'token': self.token,
|
||||
}
|
||||
|
||||
with self.client.post(
|
||||
f'{API}/compile/sessions',
|
||||
json=payload,
|
||||
name='/compile/sessions [Python interactive]',
|
||||
catch_response=True,
|
||||
timeout=30,
|
||||
) as resp:
|
||||
data = safe_json(resp)
|
||||
if resp.status_code not in (200, 202) or not data.get('session_id'):
|
||||
resp.failure(
|
||||
f"Session create failed: HTTP {resp.status_code} "
|
||||
f"{data.get('error', '')}"
|
||||
)
|
||||
return
|
||||
session_id = data['session_id']
|
||||
|
||||
# Poll sampai prompt muncul (menandakan program menunggu input)
|
||||
cursor = 0
|
||||
prompt_seen = False
|
||||
for _ in range(40): # 40 x 0.5s = 20s maks
|
||||
time.sleep(0.5)
|
||||
with self.client.get(
|
||||
f"{API}/compile/sessions/{session_id}",
|
||||
params={'cursor': cursor},
|
||||
name='/compile/sessions/{id} [poll]',
|
||||
catch_response=True,
|
||||
timeout=10,
|
||||
) as resp:
|
||||
poll = safe_json(resp)
|
||||
if resp.status_code != 200:
|
||||
resp.failure(f"Poll failed: HTTP {resp.status_code}")
|
||||
break
|
||||
cursor = poll.get('cursor', cursor)
|
||||
if 'Siapa nama kamu?' in poll.get('output', ''):
|
||||
prompt_seen = True
|
||||
break
|
||||
|
||||
if not prompt_seen:
|
||||
self.client.delete(
|
||||
f"{API}/compile/sessions/{session_id}",
|
||||
name='/compile/sessions/{id} [cleanup]',
|
||||
)
|
||||
return
|
||||
|
||||
# Kirim input (tanpa newline — worker menambahkannya)
|
||||
with self.client.post(
|
||||
f"{API}/compile/sessions/{session_id}/input",
|
||||
json={'text': 'Budi'},
|
||||
name='/compile/sessions/{id}/input',
|
||||
catch_response=True,
|
||||
timeout=10,
|
||||
) as resp:
|
||||
if resp.status_code != 200:
|
||||
resp.failure(f"Input failed: HTTP {resp.status_code}")
|
||||
|
||||
# Poll sampai selesai; verifikasi output
|
||||
for _ in range(40):
|
||||
time.sleep(0.5)
|
||||
with self.client.get(
|
||||
f"{API}/compile/sessions/{session_id}",
|
||||
params={'cursor': cursor},
|
||||
name='/compile/sessions/{id} [poll]',
|
||||
catch_response=True,
|
||||
timeout=10,
|
||||
) as resp:
|
||||
poll = safe_json(resp)
|
||||
if resp.status_code != 200:
|
||||
resp.failure(f"Poll failed: HTTP {resp.status_code}")
|
||||
break
|
||||
cursor = poll.get('cursor', cursor)
|
||||
status = poll.get('status')
|
||||
if status in ('exited', 'error', 'stopped'):
|
||||
output = poll.get('output', '')
|
||||
if 'Halo, Budi!' not in output:
|
||||
resp.failure(
|
||||
f"Output mismatch (status={status}): "
|
||||
f"{output[-120:]!r}"
|
||||
)
|
||||
break
|
||||
|
||||
# Cleanup session (best-effort)
|
||||
self.client.delete(
|
||||
f"{API}/compile/sessions/{session_id}",
|
||||
name='/compile/sessions/{id} [cleanup]',
|
||||
)
|
||||
|
||||
# ── Task 5: Verify Arduino Lesson Structure (weight=2) ─────────────
|
||||
|
||||
@task(2)
|
||||
|
|
|
|||
2
velxio
2
velxio
|
|
@ -1 +1 @@
|
|||
Subproject commit 6e1781926e5988ac5eb6ea51a30a4e8f945beec7
|
||||
Subproject commit 2b1747041b06a67db8d758908fd54d516af86927
|
||||
Loading…
Reference in New Issue