2 -description: Live-test a Pi extension in observable terminal/tmux with real Pi
3 -argument-hint: "<
extension-or-bundle> :: <
scenario/use-case>"
4 -tools: [read, write, edit, bash, grep, find, ls]
2 +description: Interactively test a Pi extension in a human-visible terminal
3 +argument-hint: "<
extension> :: <
scenario>"
8 6 #
Live Pi Extension Test 10 -Test this Pi extension/use case like a human coding-harness engineer:
8 +Test through real interactive Pi:
18 -- Open a fresh observable terminal split/window.
19 -- In it run a dedicated tmux server with exactly one tmux session and exactly
21 -- Use as many panes as needed inside that one window.
22 -- Drive real interactive `pi`, with real LLM models.
23 -- Prefer default configured model for main run.
24 - Do not use mocks/fake providers unless scenario explicitly demands it.
25 -- Load only the extension under test by default.
26 - Load support extensions only when the scenario requires their real UX
28 -- Act like a human using Pi, not like a unit-test harness.
29 -- Watch for bugs, perf issues, awkward UI/UX, confusing states, bad messages,
30 - lifecycle/reload bugs, visual glitches, terminal rendering issues, browser UI
31 - issues, and model/tool behavior problems.
32 -- Probe the user's active shell before sending commands; many users run
33 - Nushell/Fish/PowerShell, not Bash.
34 -- If an extension opens HTML/browser UI, treat that browser UI as part of the
35 - product and test it with Chrome CDP or Firefox BiDi skills.
36 -- Be brutal, skeptical, and distrustful of the Pi agent under test.
37 -- Never accept the nested Pi agent's claims, status text, final answer, or
38 - self-reported verification at face value.
39 -- Keep running notes in a temp markdown file while testing.
40 -- Finish with a concise findings report and path to the notes file.
42 -##
Coding-Harness Engineer Mandate 44 -You are not merely a user of Pi — you are a **coding-harness engineer**.
45 -Your job is to actively engineer the harness around Pi so the extension under
46 -test performs at its best.
49 -- **Seek the gold hand.** Before launching Pi, think:
50 - what would the ideal test environment look like for this extension?
52 -- **Use project-local Pi modifications.** Scan the repo for local skills
53 - (`./skills/`), local prompts (`./.pi/prompts/`), local context files
54 - (`./.pi/context-files/`), local extensions (`./extensions/`), and any
55 - workspace-level `.pi/` config.
56 - If a temporary local skill, prompt template, context file, or small
57 - scaffolding extension would improve the test, create it in the project's
58 - `.pi/` tree (or appropriate local path).
59 - Clean it up after testing unless it represents a genuine improvement worth
61 -- **Load global extensions if they provide testing infrastructure.** You may
62 - load global extensions (e.g., `chrome-cdp`, `firefox-bidi`, `tmux`,
63 - `research`) when they give you better observability, setup, or control.
64 - **Never modify any global extension except the one under test.**
65 -- **Tweak the harness, not the goal.** Every modification you make — env vars,
66 - preloaded skills, scaffolded test projects, injected prompts, context files —
67 - must serve the single purpose of improving the extension under test.
68 - Do not drift into generic harness polishing.
69 -- **Remember your true north.** Even though you are pretending to be a
70 - coding-harness engineer, the ultimate goal is to improve the extension under
72 - If a harness change reveals a bug, missing feature, or UX flaw in the
73 - extension, file it as a finding and, when appropriate, patch the extension
78 -Extract from `$ARGUMENTS`:
80 -- target extension(s)
81 -- support extension(s), if any
82 -- exact user workflow/use case
83 -- project/cwd to test in
87 -If target is ambiguous, inspect repo structure and choose the most likely
89 -If still ambiguous, ask one focused question.
93 -Load these skills when their domain is active:
95 -- **`tmux`** — for all session/pane creation, `send-keys`, `capture-pane`, and
96 - multi-pane orchestration.
97 -- **`nushell`** — when the pane shell is `nu` or `nu.exe`.
98 - Nushell has no `&&`, no `$VAR` interpolation, and `;` is the statement
100 -- **`wezterm`** — when `WEZTERM_PANE` is set and you need to split panes or
101 - attach via WezTerm CLI.
105 -Before creating or driving panes, identify the user's shell and terminal
107 -Do not assume Bash syntax inside tmux panes.
112 -USER_SHELL
="$(basename "${SHELL:-unknown}")" 113 -TERM_ENV
="WEZTERM_PANE=${WEZTERM_PANE-} GHOSTTY_RESOURCES_DIR=${GHOSTTY_RESOURCES_DIR-} KITTY_WINDOW_ID=${KITTY_WINDOW_ID-} ALACRITTY_SOCKET=${ALACRITTY_SOCKET-} TERM_PROGRAM=${TERM_PROGRAM-} WT_SESSION=${WT_SESSION-}" 114 -printf 'shell=%s\n%s\n' "$USER_SHELL" "$TERM_ENV" 120 - use `;`, not `&&`; env vars use `$env.NAME`; avoid Bash snippets in the pane.
121 - **Load the `nushell` skill** for exact syntax.
123 - use `;` or `and`, not Bash-only constructs.
125 - use `;`; env vars use `$env:NAME`.
127 - `&&`, `VAR=value cmd`, and POSIX snippets are OK.
128 -- If unsure, run complex setup from the agent's `bash` tool, not by pasting Bash
129 - into the user's pane.
130 -- When sending commands to tmux panes, prefer simple one-command lines or
131 - shell-specific forms.
133 -##
Observable Terminal + tmux Setup 135 -Load the **`tmux`** skill before writing any tmux orchestration.
137 -Create a dedicated tmux socket so the test environment has exactly one
138 -session/window without touching the user's existing tmux:
141 -RUN_ID
="pi-ext-$(date +%Y%m%d-%H%M%S)" 144 -NOTES
="/tmp/$RUN_ID-findings.md" 146 -USER_SHELL
="$(basename "${SHELL:-unknown}")" 148 -cat
> "$NOTES" <<EOF 149 -# Live Pi Extension Test 153 -- scenario: $ARGUMENTS 154 -- started: $(date -Is) 155 -- shell: $USER_SHELL 161 -tmux -L
"$SOCKET" new-session -d -s
"$SESSION" -n main -c
"$ROOT" 162 -tmux -L
"$SOCKET" split-window -h -t
"$SESSION:main" -c
"$ROOT" 163 -tmux -L
"$SOCKET" split-window -v -t
"$SESSION:main.2" -c
"$ROOT" 164 -tmux -L
"$SOCKET" select-pane -t
"$SESSION:main.1" -T
"pi interactive" 165 -tmux -L
"$SOCKET" select-pane -t
"$SESSION:main.2" -T
"notes" 166 -tmux -L
"$SOCKET" select-pane -t
"$SESSION:main.3" -T
"observer" 167 -tmux -L
"$SOCKET" send-keys -t
"$SESSION:main.2" "tail -f '$NOTES'" C-m
168 -tmux -L
"$SOCKET" send-keys -t
"$SESSION:main.3" "printf 'observer pane ready\\nnotes: $NOTES\\nshell: $USER_SHELL\\n'" C-m
169 -tmux -L
"$SOCKET" select-pane -t
"$SESSION:main.1" 171 -PANE_SHELL
="$(tmux -L "$SOCKET" display-message -p -t "$SESSION:main.1" '#{pane_current_command}')" 172 -echo "pane shell: $PANE_SHELL" >> "$NOTES" 174 -tmux -L
"$SOCKET" list-sessions
175 -tmux -L
"$SOCKET" list-windows -t
"$SESSION" 178 -Use a shell-aware pane launch command.
179 -**Keep it simple** — one command per `send-keys`, no shell operators inside the
180 -pane if Nushell is running:
183 -PI_CMD
="pi --no-extensions --no-skills --no-prompt-templates --no-context-files -e ./extensions/<target>/index.ts" 184 -case "$PANE_SHELL" in 187 - tmux -L
"$SOCKET" send-keys -t
"$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
191 - tmux -L
"$SOCKET" send-keys -t
"$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
193 - pwsh
|powershell.exe
) 195 - tmux -L
"$SOCKET" send-keys -t
"$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
198 - tmux -L
"$SOCKET" send-keys -t
"$SESSION:main.1" "cd '$ROOT' && $PI_CMD" C-m
205 -- `tmux -L "$SOCKET" list-sessions` shows exactly one session:
207 -- `tmux -L "$SOCKET" list-windows -t live` shows exactly one window:
209 -- more panes are OK; more sessions/windows are not
211 -Open/attach that tmux session in the current terminal emulator, preferring an
212 -actual split over a new OS window:
216 -Load the **`wezterm`** skill before using WezTerm CLI commands.
219 -wezterm cli split-pane --right --cwd
"$ROOT" -- tmux -L
"$SOCKET" attach -t
"$SESSION" 222 -1. Ghostty / Kitty / Alacritty / GNOME Terminal / Terminal.app / Windows
223 - Terminal / other popular terminals:
225 -- Detect via env and available commands:
226 - `WEZTERM_PANE`, `GHOSTTY_*`, `KITTY_WINDOW_ID`, `ALACRITTY_*`, `TERM_PROGRAM`,
227 - `WT_SESSION`, `TMUX`, `uname`.
228 -- Use the terminal's CLI to open a split/window attached to:
231 -tmux -L
"$SOCKET" attach -t
"$SESSION" 234 -1. If no terminal split/window can be controlled safely:
236 -- Keep the tmux server running.
237 -- Print the exact attach command prominently.
238 -- Do not continue hidden; ask the user to attach/open it.
242 -In pane 1, launch real Pi interactively.
247 -pi --no-extensions --no-skills --no-prompt-templates --no-context-files -e ./extensions/
<target
>/index.ts
250 -Adjust only when needed:
252 -- support extension:
253 - add another `-e ./extensions/<support>/index.ts`
254 -- local package extension path:
256 -- scenario requires skills/prompts/context:
257 - remove only the needed isolation flag
258 -- scenario requires previous session behavior:
259 - use normal session mode intentionally and note why
261 -Do not force a model unless scenario requires model-specific behavior.
262 -Default configured model is preferred.
263 -For very long validation, it is OK to perform one secondary run with another
264 -available real model if useful.
266 -##
Critical Verification Rules 268 -The Pi agent inside tmux is the system under test.
269 -Treat it like an unreliable junior engineer trying to impress you.
271 -- Do not trust its claims of success, failure, verification, implementation, or
273 -- Independently inspect files it says it changed.
274 -- Independently rerun commands it says passed.
275 -- Independently inspect browser/terminal screenshots for UI claims.
276 -- **Model/provider gotcha**:
277 - when an extension frontmatter says `model:
278 - openai-codex/gpt-5.5`, verify the exact provider name with `pi --list-models`
279 - and with `getModel(provider, id)` from `@earendil-works/pi-ai`.
280 - Invalid provider strings often return `undefined`, and `createAgentSession`
281 - may silently fall back to the parent session's default model.
282 -- Check whether it ignored instructions, skipped workflows, invented evidence,
283 - hid errors, or reframed the task smaller.
284 -- Record every unsupported claim as a finding until verified.
285 -- Prefer hard evidence:
286 - file diffs, command output, screenshots, browser console/network logs, session
287 - JSONL, pane captures, **and direct SDK integration tests** when the TUI is not
289 -- Be concise but harsh in notes.
290 - No praise unless evidence earned it.
292 -##
Human-like Driving Rules 294 -Interact through Pi as a human would:
296 -- type commands/prompts into pane 1
297 -- wait for UI states to settle
298 -- read terminal output
300 - Pi uses the alternate screen buffer.
301 - `tmux capture-pane` often shows only the status bar, not chat content.
302 - If you cannot see responses, verify via file-system artifacts, session JSONL,
303 - or direct SDK tests instead of fighting the TUI.
304 -- **Key concatenation gotcha**:
305 - `C-c C-u` does not reliably clear Pi's input widget.
306 - Always send `C-c`, wait, then `C-u`, wait, then type the new command.
307 - If you see merged commands like `/foo/bar`, the previous input was not
309 -- use extension slash commands, shortcuts, reloads, session switches, browser
310 - UIs, or tools only as real users would
311 -- if browser UI opens, inspect it and record UX findings with browser
312 - automation, not just terminal logs
313 -- test happy path, rough path, cancel/abort, reload/session lifecycle, browser
314 - lifecycle, and one realistic long-running path
315 -- observe latency and visual stability; note jank, flicker, unclear status,
316 - noisy widgets, bad footer/status behavior, stale state, or confusing errors
317 -- avoid unit-test shortcuts unless using them only to understand implementation
319 -This is expected to take time.
322 -Waiting is allowed only when tied to observable progress.
324 -##
Timeout + Wait Budget 326 -Use short, intelligent, condition-based waits.
327 -Do not choose huge defensive timeouts.
331 -- shell/local commands: 5–15s unless known slow
332 -- Pi startup/readiness: 15s
333 -- UI settle after a keypress: 0.2–1s
334 -- browser navigation/DOM readiness: 10–15s
335 -- ordinary build/test/probe: 30s
336 -- nested model no-progress limit: 60s, then capture evidence and interrupt/escalate
340 -- Prefer polling `pane_current_command`, files, session JSONL, process state, DOM state, or network state over sleeping.
341 -- Check progress every 5–10s for long model/tool runs.
342 -- Any timeout >30s must be written in notes with reason, expected progress signal, and stop condition.
343 -- Any timeout >120s needs explicit scenario justification; otherwise it is a harness bug.
344 -- If blocked, fail fast with evidence instead of waiting minutes silently.
346 -##
Browser UI Testing 348 -Some Pi extensions open local HTML pages.
349 -If a page opens or a URL is printed, that page is part of the extension UX and
354 -1. Detect URL/browser from terminal output, server info files, `xdg-open`/`open`
355 - behavior, or extension docs.
356 -2. If Chrome/Chromium is open or appropriate, load the `chrome-cdp` skill and
357 - use it to inspect/drive the live page.
358 -3. If Firefox is open or appropriate, load the `firefox-bidi` skill and use it
359 - to inspect/drive the live page; launching a managed Firefox is OK when
361 -4. Interact like a human:
362 - click controls, type fields, resize if relevant, reload, close/reopen, test
363 - stale sessions, and verify events flow back to Pi.
364 -5. Inspect console errors, failed network requests, DOM state, accessibility
365 - labels where practical, and visual layout.
366 -6. Record browser findings in the notes with URL, browser, steps,
367 - expected/actual, and evidence.
369 -Do not skip browser testing because terminal behavior passed.
370 -If browser automation is unavailable, mark the browser portion blocked and say
371 -exactly what is missing.
375 -Append findings while testing:
378 -cat
>> "$NOTES" <<'EOF' 380 -### <timestamp> <area> 386 -- severity: low|medium|high 393 -- command used to launch Pi
394 -- exact extension paths loaded
395 -- model/provider observed when visible
396 -- terminal emulator detected
397 -- user shell and tmux pane shell detected
398 -- tmux socket/session/window
399 -- browser used, URL tested, console/network issues, and browser skill used when
401 -- screenshots/GIF paths only if created
402 -- every bug/UX issue with reproduction steps
403 -- non-bugs worth preserving
405 -##
Completion Report 409 -1. Do not kill the tmux server automatically.
410 -2. Leave tmux running for user inspection.
411 -3. Give attach and kill commands.
413 - `Do you want me to close the tmux session now?`
414 -5. Only run `tmux -L <socket> kill-server` after the user says yes.
415 -6. Ensure notes file has final summary.
419 -LIVE_EXTENSION_TEST_DONE
420 -notes: /tmp/<file>.md
421 -tmux attach: tmux -L <socket> attach -t live
422 -tmux kill: tmux -L <socket> kill-server
423 -loaded extensions: ...
425 -shell: user=<shell> pane=<shell>
426 -browser: none|chrome-cdp|firefox-bidi|blocked
427 -result: pass|issues found|blocked
429 -- [severity] title — evidence/repro
432 -If blocked by terminal control, auth/model config, missing extension path, or
433 -unsafe destructive scenario, stop and ask for the smallest missing input.
434 -If testing completes successfully, still leave tmux alive and ask whether to
437 -##
Fallback When TUI Driving Fails 439 -If `tmux capture-pane` cannot read Pi's chat output (alternate-screen-buffer
440 -issue), do not abandon the test.
441 -Fall back in this order:
443 -1. **File-system artifacts** — check extension state dirs, session JSONL, or any
444 - files the extension writes.
445 -2. **Direct SDK integration tests** — import the extension's code, call its
446 - internal helpers, or create real `createAgentSession` calls with real models
447 - to exercise the same code paths without the TUI.
448 -3. **Process inspection** — `ls -l /proc/<pid>/fd/` and `ps` to confirm the
449 - process is alive and which files it holds open.
450 -4. **Session file forensics** — read Pi's JSONL session files with `jq` to
451 - reconstruct what happened.
453 -These fallbacks are not "cheating" — they are the harness engineer doing their
454 -job when the terminal layer is opaque.
14 +- Load `terminal-harness` and follow it for terminal setup, interaction, observation, and cleanup.
15 +- Prefer provider CLI/API for pane input and capture.
16 + Reserve desktop-driving for terminal-host bindings, mouse/focus behavior, or visual-only evidence.
17 +- Resolve the target extension, working directory, workflow, and success criteria.
18 + Ask one focused question only when ambiguity blocks testing.
19 +- Start a fresh Pi with the target extension and default configured real model.
20 + Exclude unrelated extensions; load support resources only when the workflow needs them.
21 +- Drive Pi as a user would: enter prompts and commands, press keys, wait for visible state changes, and inspect each resulting screen.
22 +- Treat terminal and browser UI as product behavior.
23 + If the extension opens browser UI, use the appropriate browser-driving skill.
24 +- Exercise the requested workflow plus relevant failure, cancellation, reload, and session behavior.
25 +- Do not replace interactive observation with unit tests, session files, or the nested Pi agent's claims.
26 + Use those only for independent verification or when recording a harness limitation.
27 +- Record defects with reproduction steps, expected behavior, actual behavior, severity, and evidence.
28 + Separate product defects from harness failures.
29 +- Finish with `pass`, `issues found`, or `blocked`, followed by concise findings and terminal inspection or cleanup information.