repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

test: simplify live extension harness prompt

44af8d33297dab76ba299800afd1c666b98200b1

Oliver Krylow <o.krylow@isp-insoft.de> ·

SSH signature · not checked

Browse filesCompare with HEAD

Changed files (1)

.pi/prompts/live-extension-test.md

Modified
2 -description: Live-test a Pi extension in observable terminal/tmux with real Pi
3 -argument-hint: "<extension-or-bundle> :: <scenario/use-case>"
4 -tools: [read, write, edit, bash, grep, find, ls]
5 -thinking: high
2 +description: Interactively test a Pi extension in a human-visible terminal
3 +argument-hint: "<extension> :: <scenario>"
8 6 # Live Pi Extension Test
10 -Test this Pi extension/use case like a human coding-harness engineer:
8 +Test through real interactive Pi:
12 10 ```text
13 11 $ARGUMENTS
16 -Mission:
18 -- Open a fresh observable terminal split/window.
19 -- In it run a dedicated tmux server with exactly one tmux session and exactly
20 - one tmux window.
21 -- Use as many panes as needed inside that one window.
22 -- Drive real interactive `pi`, with real LLM models.
23 -- Prefer default configured model for main run.
24 - Do not use mocks/fake providers unless scenario explicitly demands it.
25 -- Load only the extension under test by default.
26 - Load support extensions only when the scenario requires their real UX
27 - interaction.
28 -- Act like a human using Pi, not like a unit-test harness.
29 -- Watch for bugs, perf issues, awkward UI/UX, confusing states, bad messages,
30 - lifecycle/reload bugs, visual glitches, terminal rendering issues, browser UI
31 - issues, and model/tool behavior problems.
32 -- Probe the user's active shell before sending commands; many users run
33 - Nushell/Fish/PowerShell, not Bash.
34 -- If an extension opens HTML/browser UI, treat that browser UI as part of the
35 - product and test it with Chrome CDP or Firefox BiDi skills.
36 -- Be brutal, skeptical, and distrustful of the Pi agent under test.
37 -- Never accept the nested Pi agent's claims, status text, final answer, or
38 - self-reported verification at face value.
39 -- Keep running notes in a temp markdown file while testing.
40 -- Finish with a concise findings report and path to the notes file.
42 -## Coding-Harness Engineer Mandate
44 -You are not merely a user of Pi — you are a **coding-harness engineer**.
45 -Your job is to actively engineer the harness around Pi so the extension under
46 -test performs at its best.
47 -This means:
49 -- **Seek the gold hand.** Before launching Pi, think:
50 - what would the ideal test environment look like for this extension?
51 - Then build it.
52 -- **Use project-local Pi modifications.** Scan the repo for local skills
53 - (`./skills/`), local prompts (`./.pi/prompts/`), local context files
54 - (`./.pi/context-files/`), local extensions (`./extensions/`), and any
55 - workspace-level `.pi/` config.
56 - If a temporary local skill, prompt template, context file, or small
57 - scaffolding extension would improve the test, create it in the project's
58 - `.pi/` tree (or appropriate local path).
59 - Clean it up after testing unless it represents a genuine improvement worth
60 - keeping.
61 -- **Load global extensions if they provide testing infrastructure.** You may
62 - load global extensions (e.g., `chrome-cdp`, `firefox-bidi`, `tmux`,
63 - `research`) when they give you better observability, setup, or control.
64 - **Never modify any global extension except the one under test.**
65 -- **Tweak the harness, not the goal.** Every modification you make — env vars,
66 - preloaded skills, scaffolded test projects, injected prompts, context files —
67 - must serve the single purpose of improving the extension under test.
68 - Do not drift into generic harness polishing.
69 -- **Remember your true north.** Even though you are pretending to be a
70 - coding-harness engineer, the ultimate goal is to improve the extension under
71 - test.
72 - If a harness change reveals a bug, missing feature, or UX flaw in the
73 - extension, file it as a finding and, when appropriate, patch the extension
74 - itself.
76 -## Parse Scenario
78 -Extract from `$ARGUMENTS`:
80 -- target extension(s)
81 -- support extension(s), if any
82 -- exact user workflow/use case
83 -- project/cwd to test in
84 -- success criteria
85 -- risks to watch
87 -If target is ambiguous, inspect repo structure and choose the most likely
88 -extension path.
89 -If still ambiguous, ask one focused question.
91 -## Skills to Load
93 -Load these skills when their domain is active:
95 -- **`tmux`** — for all session/pane creation, `send-keys`, `capture-pane`, and
96 - multi-pane orchestration.
97 -- **`nushell`** — when the pane shell is `nu` or `nu.exe`.
98 - Nushell has no `&&`, no `$VAR` interpolation, and `;` is the statement
99 - separator.
100 -- **`wezterm`** — when `WEZTERM_PANE` is set and you need to split panes or
101 - attach via WezTerm CLI.
103 -## Shell Probe
105 -Before creating or driving panes, identify the user's shell and terminal
106 -environment.
107 -Do not assume Bash syntax inside tmux panes.
109 -Probe:
111 -```sh
112 -USER_SHELL="$(basename "${SHELL:-unknown}")"
113 -TERM_ENV="WEZTERM_PANE=${WEZTERM_PANE-} GHOSTTY_RESOURCES_DIR=${GHOSTTY_RESOURCES_DIR-} KITTY_WINDOW_ID=${KITTY_WINDOW_ID-} ALACRITTY_SOCKET=${ALACRITTY_SOCKET-} TERM_PROGRAM=${TERM_PROGRAM-} WT_SESSION=${WT_SESSION-}"
114 -printf 'shell=%s\n%s\n' "$USER_SHELL" "$TERM_ENV"
115 -```
117 -Shell rules:
119 -- Nushell:
120 - use `;`, not `&&`; env vars use `$env.NAME`; avoid Bash snippets in the pane.
121 - **Load the `nushell` skill** for exact syntax.
122 -- Fish:
123 - use `;` or `and`, not Bash-only constructs.
124 -- PowerShell:
125 - use `;`; env vars use `$env:NAME`.
126 -- Bash/Zsh/Sh:
127 - `&&`, `VAR=value cmd`, and POSIX snippets are OK.
128 -- If unsure, run complex setup from the agent's `bash` tool, not by pasting Bash
129 - into the user's pane.
130 -- When sending commands to tmux panes, prefer simple one-command lines or
131 - shell-specific forms.
133 -## Observable Terminal + tmux Setup
135 -Load the **`tmux`** skill before writing any tmux orchestration.
137 -Create a dedicated tmux socket so the test environment has exactly one
138 -session/window without touching the user's existing tmux:
140 -```sh
141 -RUN_ID="pi-ext-$(date +%Y%m%d-%H%M%S)"
142 -SOCKET="$RUN_ID"
143 -SESSION="live"
144 -NOTES="/tmp/$RUN_ID-findings.md"
145 -ROOT="$PWD"
146 -USER_SHELL="$(basename "${SHELL:-unknown}")"
148 -cat > "$NOTES" <<EOF
149 -# Live Pi Extension Test
151 -- run: $RUN_ID
152 -- cwd: $ROOT
153 -- scenario: $ARGUMENTS
154 -- started: $(date -Is)
155 -- shell: $USER_SHELL
157 -## Timeline
159 -EOF
161 -tmux -L "$SOCKET" new-session -d -s "$SESSION" -n main -c "$ROOT"
162 -tmux -L "$SOCKET" split-window -h -t "$SESSION:main" -c "$ROOT"
163 -tmux -L "$SOCKET" split-window -v -t "$SESSION:main.2" -c "$ROOT"
164 -tmux -L "$SOCKET" select-pane -t "$SESSION:main.1" -T "pi interactive"
165 -tmux -L "$SOCKET" select-pane -t "$SESSION:main.2" -T "notes"
166 -tmux -L "$SOCKET" select-pane -t "$SESSION:main.3" -T "observer"
167 -tmux -L "$SOCKET" send-keys -t "$SESSION:main.2" "tail -f '$NOTES'" C-m
168 -tmux -L "$SOCKET" send-keys -t "$SESSION:main.3" "printf 'observer pane ready\\nnotes: $NOTES\\nshell: $USER_SHELL\\n'" C-m
169 -tmux -L "$SOCKET" select-pane -t "$SESSION:main.1"
171 -PANE_SHELL="$(tmux -L "$SOCKET" display-message -p -t "$SESSION:main.1" '#{pane_current_command}')"
172 -echo "pane shell: $PANE_SHELL" >> "$NOTES"
174 -tmux -L "$SOCKET" list-sessions
175 -tmux -L "$SOCKET" list-windows -t "$SESSION"
176 -```
178 -Use a shell-aware pane launch command.
179 -**Keep it simple** — one command per `send-keys`, no shell operators inside the
180 -pane if Nushell is running:
182 -```sh
183 -PI_CMD="pi --no-extensions --no-skills --no-prompt-templates --no-context-files -e ./extensions/<target>/index.ts"
184 -case "$PANE_SHELL" in
185 - nu|nu.exe)
186 - # Nushell: ';' is the statement separator. '&&' does not exist.
187 - tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
188 - ;;
189 - fish)
190 - # Fish: ';' works, 'and' is the keyword form.
191 - tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
192 - ;;
193 - pwsh|powershell.exe)
194 - # PowerShell: ';' is the statement separator.
195 - tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
196 - ;;
197 - *)
198 - tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT' && $PI_CMD" C-m
199 - ;;
200 -esac
201 -```
203 -Invariant:
205 -- `tmux -L "$SOCKET" list-sessions` shows exactly one session:
206 - `live`
207 -- `tmux -L "$SOCKET" list-windows -t live` shows exactly one window:
208 - `main`
209 -- more panes are OK; more sessions/windows are not
211 -Open/attach that tmux session in the current terminal emulator, preferring an
212 -actual split over a new OS window:
214 -1. WezTerm:
216 -Load the **`wezterm`** skill before using WezTerm CLI commands.
218 -```sh
219 -wezterm cli split-pane --right --cwd "$ROOT" -- tmux -L "$SOCKET" attach -t "$SESSION"
220 -```
222 -1. Ghostty / Kitty / Alacritty / GNOME Terminal / Terminal.app / Windows
223 - Terminal / other popular terminals:
225 -- Detect via env and available commands:
226 - `WEZTERM_PANE`, `GHOSTTY_*`, `KITTY_WINDOW_ID`, `ALACRITTY_*`, `TERM_PROGRAM`,
227 - `WT_SESSION`, `TMUX`, `uname`.
228 -- Use the terminal's CLI to open a split/window attached to:
230 -```sh
231 -tmux -L "$SOCKET" attach -t "$SESSION"
232 -```
234 -1. If no terminal split/window can be controlled safely:
236 -- Keep the tmux server running.
237 -- Print the exact attach command prominently.
238 -- Do not continue hidden; ask the user to attach/open it.
240 -## Start Pi
242 -In pane 1, launch real Pi interactively.
244 -Default isolation:
246 -```sh
247 -pi --no-extensions --no-skills --no-prompt-templates --no-context-files -e ./extensions/<target>/index.ts
248 -```
250 -Adjust only when needed:
252 -- support extension:
253 - add another `-e ./extensions/<support>/index.ts`
254 -- local package extension path:
255 - use exact path
256 -- scenario requires skills/prompts/context:
257 - remove only the needed isolation flag
258 -- scenario requires previous session behavior:
259 - use normal session mode intentionally and note why
261 -Do not force a model unless scenario requires model-specific behavior.
262 -Default configured model is preferred.
263 -For very long validation, it is OK to perform one secondary run with another
264 -available real model if useful.
266 -## Critical Verification Rules
268 -The Pi agent inside tmux is the system under test.
269 -Treat it like an unreliable junior engineer trying to impress you.
271 -- Do not trust its claims of success, failure, verification, implementation, or
272 - UX quality.
273 -- Independently inspect files it says it changed.
274 -- Independently rerun commands it says passed.
275 -- Independently inspect browser/terminal screenshots for UI claims.
276 -- **Model/provider gotcha**:
277 - when an extension frontmatter says `model:
278 - openai-codex/gpt-5.5`, verify the exact provider name with `pi --list-models`
279 - and with `getModel(provider, id)` from `@earendil-works/pi-ai`.
280 - Invalid provider strings often return `undefined`, and `createAgentSession`
281 - may silently fall back to the parent session's default model.
282 -- Check whether it ignored instructions, skipped workflows, invented evidence,
283 - hid errors, or reframed the task smaller.
284 -- Record every unsupported claim as a finding until verified.
285 -- Prefer hard evidence:
286 - file diffs, command output, screenshots, browser console/network logs, session
287 - JSONL, pane captures, **and direct SDK integration tests** when the TUI is not
288 - observable.
289 -- Be concise but harsh in notes.
290 - No praise unless evidence earned it.
292 -## Human-like Driving Rules
294 -Interact through Pi as a human would:
296 -- type commands/prompts into pane 1
297 -- wait for UI states to settle
298 -- read terminal output
299 -- **TUI gotcha**:
300 - Pi uses the alternate screen buffer.
301 - `tmux capture-pane` often shows only the status bar, not chat content.
302 - If you cannot see responses, verify via file-system artifacts, session JSONL,
303 - or direct SDK tests instead of fighting the TUI.
304 -- **Key concatenation gotcha**:
305 - `C-c C-u` does not reliably clear Pi's input widget.
306 - Always send `C-c`, wait, then `C-u`, wait, then type the new command.
307 - If you see merged commands like `/foo/bar`, the previous input was not
308 - cleared.
309 -- use extension slash commands, shortcuts, reloads, session switches, browser
310 - UIs, or tools only as real users would
311 -- if browser UI opens, inspect it and record UX findings with browser
312 - automation, not just terminal logs
313 -- test happy path, rough path, cancel/abort, reload/session lifecycle, browser
314 - lifecycle, and one realistic long-running path
315 -- observe latency and visual stability; note jank, flicker, unclear status,
316 - noisy widgets, bad footer/status behavior, stale state, or confusing errors
317 -- avoid unit-test shortcuts unless using them only to understand implementation
319 -This is expected to take time.
320 -Be thorough.
321 -Thorough ≠ idle.
322 -Waiting is allowed only when tied to observable progress.
324 -## Timeout + Wait Budget
326 -Use short, intelligent, condition-based waits.
327 -Do not choose huge defensive timeouts.
329 -Defaults:
331 -- shell/local commands: 5–15s unless known slow
332 -- Pi startup/readiness: 15s
333 -- UI settle after a keypress: 0.2–1s
334 -- browser navigation/DOM readiness: 10–15s
335 -- ordinary build/test/probe: 30s
336 -- nested model no-progress limit: 60s, then capture evidence and interrupt/escalate
338 -Rules:
340 -- Prefer polling `pane_current_command`, files, session JSONL, process state, DOM state, or network state over sleeping.
341 -- Check progress every 5–10s for long model/tool runs.
342 -- Any timeout >30s must be written in notes with reason, expected progress signal, and stop condition.
343 -- Any timeout >120s needs explicit scenario justification; otherwise it is a harness bug.
344 -- If blocked, fail fast with evidence instead of waiting minutes silently.
346 -## Browser UI Testing
348 -Some Pi extensions open local HTML pages.
349 -If a page opens or a URL is printed, that page is part of the extension UX and
350 -must be tested.
352 -Browser protocol:
354 -1. Detect URL/browser from terminal output, server info files, `xdg-open`/`open`
355 - behavior, or extension docs.
356 -2. If Chrome/Chromium is open or appropriate, load the `chrome-cdp` skill and
357 - use it to inspect/drive the live page.
358 -3. If Firefox is open or appropriate, load the `firefox-bidi` skill and use it
359 - to inspect/drive the live page; launching a managed Firefox is OK when
360 - needed.
361 -4. Interact like a human:
362 - click controls, type fields, resize if relevant, reload, close/reopen, test
363 - stale sessions, and verify events flow back to Pi.
364 -5. Inspect console errors, failed network requests, DOM state, accessibility
365 - labels where practical, and visual layout.
366 -6. Record browser findings in the notes with URL, browser, steps,
367 - expected/actual, and evidence.
369 -Do not skip browser testing because terminal behavior passed.
370 -If browser automation is unavailable, mark the browser portion blocked and say
371 -exactly what is missing.
373 -## Notes Discipline
375 -Append findings while testing:
377 -```sh
378 -cat >> "$NOTES" <<'EOF'
380 -### <timestamp> <area>
382 -- observation:
383 -- steps:
384 -- expected:
385 -- actual:
386 -- severity: low|medium|high
387 -- evidence:
388 -EOF
389 -```
391 -Record:
393 -- command used to launch Pi
394 -- exact extension paths loaded
395 -- model/provider observed when visible
396 -- terminal emulator detected
397 -- user shell and tmux pane shell detected
398 -- tmux socket/session/window
399 -- browser used, URL tested, console/network issues, and browser skill used when
400 - applicable
401 -- screenshots/GIF paths only if created
402 -- every bug/UX issue with reproduction steps
403 -- non-bugs worth preserving
405 -## Completion Report
407 -When done:
409 -1. Do not kill the tmux server automatically.
410 -2. Leave tmux running for user inspection.
411 -3. Give attach and kill commands.
412 -4. Ask:
413 - `Do you want me to close the tmux session now?`
414 -5. Only run `tmux -L <socket> kill-server` after the user says yes.
415 -6. Ensure notes file has final summary.
416 -7. Report:
418 -```text
419 -LIVE_EXTENSION_TEST_DONE
420 -notes: /tmp/<file>.md
421 -tmux attach: tmux -L <socket> attach -t live
422 -tmux kill: tmux -L <socket> kill-server
423 -loaded extensions: ...
424 -model: default|...
425 -shell: user=<shell> pane=<shell>
426 -browser: none|chrome-cdp|firefox-bidi|blocked
427 -result: pass|issues found|blocked
428 -findings:
429 -- [severity] title — evidence/repro
430 -```
432 -If blocked by terminal control, auth/model config, missing extension path, or
433 -unsafe destructive scenario, stop and ask for the smallest missing input.
434 -If testing completes successfully, still leave tmux alive and ask whether to
435 -close it.
437 -## Fallback When TUI Driving Fails
439 -If `tmux capture-pane` cannot read Pi's chat output (alternate-screen-buffer
440 -issue), do not abandon the test.
441 -Fall back in this order:
443 -1. **File-system artifacts** — check extension state dirs, session JSONL, or any
444 - files the extension writes.
445 -2. **Direct SDK integration tests** — import the extension's code, call its
446 - internal helpers, or create real `createAgentSession` calls with real models
447 - to exercise the same code paths without the TUI.
448 -3. **Process inspection** — `ls -l /proc/<pid>/fd/` and `ps` to confirm the
449 - process is alive and which files it holds open.
450 -4. **Session file forensics** — read Pi's JSONL session files with `jq` to
451 - reconstruct what happened.
453 -These fallbacks are not "cheating" — they are the harness engineer doing their
454 -job when the terminal layer is opaque.
14 +- Load `terminal-harness` and follow it for terminal setup, interaction, observation, and cleanup.
15 +- Prefer provider CLI/API for pane input and capture.
16 + Reserve desktop-driving for terminal-host bindings, mouse/focus behavior, or visual-only evidence.
17 +- Resolve the target extension, working directory, workflow, and success criteria.
18 + Ask one focused question only when ambiguity blocks testing.
19 +- Start a fresh Pi with the target extension and default configured real model.
20 + Exclude unrelated extensions; load support resources only when the workflow needs them.
21 +- Drive Pi as a user would: enter prompts and commands, press keys, wait for visible state changes, and inspect each resulting screen.
22 +- Treat terminal and browser UI as product behavior.
23 + If the extension opens browser UI, use the appropriate browser-driving skill.
24 +- Exercise the requested workflow plus relevant failure, cancellation, reload, and session behavior.
25 +- Do not replace interactive observation with unit tests, session files, or the nested Pi agent's claims.
26 + Use those only for independent verification or when recording a harness limitation.
27 +- Record defects with reproduction steps, expected behavior, actual behavior, severity, and evidence.
28 + Separate product defects from harness failures.
29 +- Finish with `pass`, `issues found`, or `blocked`, followed by concise findings and terminal inspection or cleanup information.