2 +description: Live-test a Pi extension in observable terminal/tmux with real Pi
3 +argument-hint: "<
extension-or-bundle> :: <
scenario/use-case>"
4 +tools: [read, write, edit, bash, grep, find, ls]
8 +#
Live Pi Extension Test 10 +Test this Pi extension/use case like a human coding-harness engineer:
18 +- Open a fresh observable terminal split/window.
19 +- In it run a dedicated tmux server with exactly one tmux session and exactly one tmux window.
20 +- Use as many panes as needed inside that one window.
21 +- Drive real interactive `pi`, with real LLM models.
22 +- Prefer default configured model for main run. Do not use mocks/fake providers unless scenario explicitly demands it.
23 +- Load only the extension under test by default. Load support extensions only when the scenario requires their real UX interaction.
24 +- Act like a human using Pi, not like a unit-test harness.
25 +- Watch for bugs, perf issues, awkward UI/UX, confusing states, bad messages, lifecycle/reload bugs, visual glitches, terminal rendering issues, browser UI issues, and model/tool behavior problems.
26 +- Probe the user's active shell before sending commands; many users run Nushell/Fish/PowerShell, not Bash.
27 +- If an extension opens HTML/browser UI, treat that browser UI as part of the product and test it with Chrome CDP or Firefox BiDi skills.
28 +- Be brutal, skeptical, and distrustful of the Pi agent under test.
29 +- Never accept the nested Pi agent's claims, status text, final answer, or self-reported verification at face value.
30 +- Keep running notes in a temp markdown file while testing.
31 +- Finish with a concise findings report and path to the notes file.
35 +Extract from `$ARGUMENTS`:
37 +- target extension(s)
38 +- support extension(s), if any
39 +- exact user workflow/use case
40 +- project/cwd to test in
44 +If target is ambiguous, inspect repo structure and choose the most likely extension path. If still ambiguous, ask one focused question.
48 +Before creating or driving panes, identify the user's shell and terminal environment. Do not assume Bash syntax inside tmux panes.
53 +USER_SHELL
="$(basename "${SHELL:-unknown}")" 54 +TERM_ENV
="WEZTERM_PANE=${WEZTERM_PANE-} GHOSTTY_RESOURCES_DIR=${GHOSTTY_RESOURCES_DIR-} KITTY_WINDOW_ID=${KITTY_WINDOW_ID-} ALACRITTY_SOCKET=${ALACRITTY_SOCKET-} TERM_PROGRAM=${TERM_PROGRAM-} WT_SESSION=${WT_SESSION-}" 55 +printf 'shell=%s\n%s\n' "$USER_SHELL" "$TERM_ENV" 60 +- Nushell: use `;`, not `&&`; env vars use `$env.NAME`; avoid Bash snippets in the pane.
61 +- Fish: use `;` or `and`, not Bash-only constructs.
62 +- PowerShell: use `;`; env vars use `$env:NAME`.
63 +- Bash/Zsh/Sh: `&&`, `VAR=value cmd`, and POSIX snippets are OK.
64 +- If unsure, run complex setup from the agent's `bash` tool, not by pasting Bash into the user's pane.
65 +- When sending commands to tmux panes, prefer simple one-command lines or shell-specific forms.
67 +##
Observable Terminal + tmux Setup 69 +Create a dedicated tmux socket so the test environment has exactly one session/window without touching the user's existing tmux:
72 +RUN_ID
="pi-ext-$(date +%Y%m%d-%H%M%S)" 75 +NOTES
="/tmp/$RUN_ID-findings.md" 77 +USER_SHELL
="$(basename "${SHELL:-unknown}")" 80 +# Live Pi Extension Test 84 +- scenario: $ARGUMENTS 85 +- started: $(date -Is) 92 +tmux -L
"$SOCKET" new-session -d -s
"$SESSION" -n main -c
"$ROOT" 93 +tmux -L
"$SOCKET" split-window -h -t
"$SESSION:main" -c
"$ROOT" 94 +tmux -L
"$SOCKET" split-window -v -t
"$SESSION:main.2" -c
"$ROOT" 95 +tmux -L
"$SOCKET" select-pane -t
"$SESSION:main.1" -T
"pi interactive" 96 +tmux -L
"$SOCKET" select-pane -t
"$SESSION:main.2" -T
"notes" 97 +tmux -L
"$SOCKET" select-pane -t
"$SESSION:main.3" -T
"observer" 98 +tmux -L
"$SOCKET" send-keys -t
"$SESSION:main.2" "tail -f '$NOTES'" C-m
99 +tmux -L
"$SOCKET" send-keys -t
"$SESSION:main.3" "printf 'observer pane ready\\nnotes: $NOTES\\nshell: $USER_SHELL\\n'" C-m
100 +tmux -L
"$SOCKET" select-pane -t
"$SESSION:main.1" 102 +PANE_SHELL
="$(tmux -L "$SOCKET" display-message -p -t "$SESSION:main.1" '#{pane_current_command}')" 103 +echo "pane shell: $PANE_SHELL" >> "$NOTES" 105 +tmux -L
"$SOCKET" list-sessions
106 +tmux -L
"$SOCKET" list-windows -t
"$SESSION" 109 +Use a shell-aware pane launch command:
112 +PI_CMD
="pi --no-extensions --no-skills --no-prompt-templates --no-context-files -e ./extensions/<target>/index.ts" 113 +case "$PANE_SHELL" in 114 + nu
|nu.exe
|fish
|pwsh
|powershell.exe
) PANE_LAUNCH
="cd '$ROOT'; $PI_CMD" ;;
115 + *) PANE_LAUNCH
="cd '$ROOT' && $PI_CMD" ;;
118 +tmux -L
"$SOCKET" send-keys -t
"$SESSION:main.1" "$PANE_LAUNCH" C-m
123 +- `tmux -L "$SOCKET" list-sessions` shows exactly one session: `live`
124 +- `tmux -L "$SOCKET" list-windows -t live` shows exactly one window: `main`
125 +- more panes are OK; more sessions/windows are not
127 +Open/attach that tmux session in the current terminal emulator, preferring an actual split over a new OS window:
132 +wezterm cli split-pane --right --cwd
"$ROOT" -- tmux -L
"$SOCKET" attach -t
"$SESSION" 135 +2. Ghostty / Kitty / Alacritty / GNOME Terminal / Terminal.app / Windows Terminal / other popular terminals:
137 +- Detect via env and available commands: `WEZTERM_PANE`, `GHOSTTY_*`, `KITTY_WINDOW_ID`, `ALACRITTY_*`, `TERM_PROGRAM`, `WT_SESSION`, `TMUX`, `uname`.
138 +- Use the terminal's CLI to open a split/window attached to:
141 +tmux -L
"$SOCKET" attach -t
"$SESSION" 144 +3. If no terminal split/window can be controlled safely:
146 +- Keep the tmux server running.
147 +- Print the exact attach command prominently.
148 +- Do not continue hidden; ask the user to attach/open it.
152 +In pane 1, launch real Pi interactively.
157 +pi --no-extensions --no-skills --no-prompt-templates --no-context-files -e ./extensions/
<target
>/index.ts
160 +Adjust only when needed:
162 +- support extension: add another `-e ./extensions/<support>/index.ts`
163 +- local package extension path: use exact path
164 +- scenario requires skills/prompts/context: remove only the needed isolation flag
165 +- scenario requires previous session behavior: use normal session mode intentionally and note why
167 +Do not force a model unless scenario requires model-specific behavior. Default configured model is preferred. For very long validation, it is OK to perform one secondary run with another available real model if useful.
169 +##
Critical Verification Rules 171 +The Pi agent inside tmux is the system under test. Treat it like an unreliable junior engineer trying to impress you.
173 +- Do not trust its claims of success, failure, verification, implementation, or UX quality.
174 +- Independently inspect files it says it changed.
175 +- Independently rerun commands it says passed.
176 +- Independently inspect browser/terminal screenshots for UI claims.
177 +- Check whether it ignored instructions, skipped workflows, invented evidence, hid errors, or reframed the task smaller.
178 +- Record every unsupported claim as a finding until verified.
179 +- Prefer hard evidence: file diffs, command output, screenshots, browser console/network logs, session JSONL, pane captures.
180 +- Be concise but harsh in notes. No praise unless evidence earned it.
182 +##
Human-like Driving Rules 184 +Interact through Pi as a human would:
186 +- type commands/prompts into pane 1
187 +- wait for UI states to settle
188 +- read terminal output
189 +- use extension slash commands, shortcuts, reloads, session switches, browser UIs, or tools only as real users would
190 +- if browser UI opens, inspect it and record UX findings with browser automation, not just terminal logs
191 +- test happy path, rough path, cancel/abort, reload/session lifecycle, browser lifecycle, and one realistic long-running path
192 +- observe latency and visual stability; note jank, flicker, unclear status, noisy widgets, bad footer/status behavior, stale state, or confusing errors
193 +- avoid unit-test shortcuts unless using them only to understand implementation
195 +This is expected to take time. Be thorough.
197 +##
Browser UI Testing 199 +Some Pi extensions open local HTML pages. If a page opens or a URL is printed, that page is part of the extension UX and must be tested.
203 +1. Detect URL/browser from terminal output, server info files, `xdg-open`/`open` behavior, or extension docs.
204 +2. If Chrome/Chromium is open or appropriate, load the `chrome-cdp` skill and use it to inspect/drive the live page.
205 +3. If Firefox is open or appropriate, load the `firefox-bidi` skill and use it to inspect/drive the live page; launching a managed Firefox is OK when needed.
206 +4. Interact like a human: click controls, type fields, resize if relevant, reload, close/reopen, test stale sessions, and verify events flow back to Pi.
207 +5. Inspect console errors, failed network requests, DOM state, accessibility labels where practical, and visual layout.
208 +6. Record browser findings in the notes with URL, browser, steps, expected/actual, and evidence.
210 +Do not skip browser testing because terminal behavior passed. If browser automation is unavailable, mark the browser portion blocked and say exactly what is missing.
214 +Append findings while testing:
217 +cat
>> "$NOTES" <<'EOF' 219 +### <timestamp> <area> 225 +- severity: low|medium|high 232 +- command used to launch Pi
233 +- exact extension paths loaded
234 +- model/provider observed when visible
235 +- terminal emulator detected
236 +- user shell and tmux pane shell detected
237 +- tmux socket/session/window
238 +- browser used, URL tested, console/network issues, and browser skill used when applicable
239 +- screenshots/GIF paths only if created
240 +- every bug/UX issue with reproduction steps
241 +- non-bugs worth preserving
243 +##
Completion Report 247 +1. Do not kill the tmux server automatically.
248 +2. Leave tmux running for user inspection.
249 +3. Give attach and kill commands.
250 +4. Ask: `Do you want me to close the tmux session now?`
251 +5. Only run `tmux -L <socket> kill-server` after the user says yes.
252 +6. Ensure notes file has final summary.
256 +LIVE_EXTENSION_TEST_DONE
257 +notes: /tmp/<file>.md
258 +tmux attach: tmux -L <socket> attach -t live
259 +tmux kill: tmux -L <socket> kill-server
260 +loaded extensions: ...
262 +shell: user=<shell> pane=<shell>
263 +browser: none|chrome-cdp|firefox-bidi|blocked
264 +result: pass|issues found|blocked
266 +- [severity] title — evidence/repro
269 +If blocked by terminal control, auth/model config, missing extension path, or unsafe destructive scenario, stop and ask for the smallest missing input. If testing completes successfully, still leave tmux alive and ask whether to close it.