18 18 - Open a fresh observable terminal split/window.
19 -- In it run a dedicated tmux server with exactly one tmux session and exactly one tmux window.
19 +- In it run a dedicated tmux server with exactly one tmux session and exactly
20 21 - Use as many panes as needed inside that one window.
21 22 - Drive real interactive `pi`, with real LLM models.
22 -- Prefer default configured model for main run. Do not use mocks/fake providers unless scenario explicitly demands it.
23 -- Load only the extension under test by default. Load support extensions only when the scenario requires their real UX interaction.
23 +- Prefer default configured model for main run.
24 + Do not use mocks/fake providers unless scenario explicitly demands it.
25 +- Load only the extension under test by default.
26 + Load support extensions only when the scenario requires their real UX
24 28 - Act like a human using Pi, not like a unit-test harness.
25 -- Watch for bugs, perf issues, awkward UI/UX, confusing states, bad messages, lifecycle/reload bugs, visual glitches, terminal rendering issues, browser UI issues, and model/tool behavior problems.
26 -- Probe the user's active shell before sending commands; many users run Nushell/Fish/PowerShell, not Bash.
27 -- If an extension opens HTML/browser UI, treat that browser UI as part of the product and test it with Chrome CDP or Firefox BiDi skills.
29 +- Watch for bugs, perf issues, awkward UI/UX, confusing states, bad messages,
30 + lifecycle/reload bugs, visual glitches, terminal rendering issues, browser UI
31 + issues, and model/tool behavior problems.
32 +- Probe the user's active shell before sending commands; many users run
33 + Nushell/Fish/PowerShell, not Bash.
34 +- If an extension opens HTML/browser UI, treat that browser UI as part of the
35 + product and test it with Chrome CDP or Firefox BiDi skills.
28 36 - Be brutal, skeptical, and distrustful of the Pi agent under test.
29 -- Never accept the nested Pi agent's claims, status text, final answer, or self-reported verification at face value.
37 +- Never accept the nested Pi agent's claims, status text, final answer, or
38 + self-reported verification at face value.
30 39 - Keep running notes in a temp markdown file while testing.
31 40 - Finish with a concise findings report and path to the notes file.
33 42 ##
Coding-Harness Engineer Mandate 35 -You are not merely a user of Pi — you are a **coding-harness engineer**. Your job is to actively engineer the harness around Pi so the extension under test performs at its best. This means:
44 +You are not merely a user of Pi — you are a **coding-harness engineer**.
45 +Your job is to actively engineer the harness around Pi so the extension under
46 +test performs at its best.
37 -- **Seek the gold hand.** Before launching Pi, think: what would the ideal test environment look like for this extension? Then build it.
38 -- **Use project-local Pi modifications.** Scan the repo for local skills (`./skills/`), local prompts (`./.pi/prompts/`), local context files (`./.pi/context-files/`), local extensions (`./extensions/`), and any workspace-level `.pi/` config. If a temporary local skill, prompt template, context file, or small scaffolding extension would improve the test, create it in the project's `.pi/` tree (or appropriate local path). Clean it up after testing unless it represents a genuine improvement worth keeping.
39 -- **Load global extensions if they provide testing infrastructure.** You may load global extensions (e.g., `chrome-cdp`, `firefox-bidi`, `tmux`, `research`) when they give you better observability, setup, or control. **Never modify any global extension except the one under test.**
40 -- **Tweak the harness, not the goal.** Every modification you make — env vars, preloaded skills, scaffolded test projects, injected prompts, context files — must serve the single purpose of improving the extension under test. Do not drift into generic harness polishing.
41 -- **Remember your true north.** Even though you are pretending to be a coding-harness engineer, the ultimate goal is to improve the extension under test. If a harness change reveals a bug, missing feature, or UX flaw in the extension, file it as a finding and, when appropriate, patch the extension itself.
49 +- **Seek the gold hand.** Before launching Pi, think:
50 + what would the ideal test environment look like for this extension?
52 +- **Use project-local Pi modifications.** Scan the repo for local skills
53 + (`./skills/`), local prompts (`./.pi/prompts/`), local context files
54 + (`./.pi/context-files/`), local extensions (`./extensions/`), and any
55 + workspace-level `.pi/` config.
56 + If a temporary local skill, prompt template, context file, or small
57 + scaffolding extension would improve the test, create it in the project's
58 + `.pi/` tree (or appropriate local path).
59 + Clean it up after testing unless it represents a genuine improvement worth
61 +- **Load global extensions if they provide testing infrastructure.** You may
62 + load global extensions (e.g., `chrome-cdp`, `firefox-bidi`, `tmux`,
63 + `research`) when they give you better observability, setup, or control.
64 + **Never modify any global extension except the one under test.**
65 +- **Tweak the harness, not the goal.** Every modification you make — env vars,
66 + preloaded skills, scaffolded test projects, injected prompts, context files —
67 + must serve the single purpose of improving the extension under test.
68 + Do not drift into generic harness polishing.
69 +- **Remember your true north.** Even though you are pretending to be a
70 + coding-harness engineer, the ultimate goal is to improve the extension under
72 + If a harness change reveals a bug, missing feature, or UX flaw in the
73 + extension, file it as a finding and, when appropriate, patch the extension
54 -If target is ambiguous, inspect repo structure and choose the most likely extension path. If still ambiguous, ask one focused question.
87 +If target is ambiguous, inspect repo structure and choose the most likely
89 +If still ambiguous, ask one focused question.
58 93 Load these skills when their domain is active:
59 -- **`tmux`** — for all session/pane creation, `send-keys`, `capture-pane`, and multi-pane orchestration.
60 -- **`nushell`** — when the pane shell is `nu` or `nu.exe`. Nushell has no `&&`, no `$VAR` interpolation, and `;` is the statement separator.
61 -- **`wezterm`** — when `WEZTERM_PANE` is set and you need to split panes or attach via WezTerm CLI.
95 +- **`tmux`** — for all session/pane creation, `send-keys`, `capture-pane`, and
96 + multi-pane orchestration.
97 +- **`nushell`** — when the pane shell is `nu` or `nu.exe`.
98 + Nushell has no `&&`, no `$VAR` interpolation, and `;` is the statement
100 +- **`wezterm`** — when `WEZTERM_PANE` is set and you need to split panes or
101 + attach via WezTerm CLI.
65 -Before creating or driving panes, identify the user's shell and terminal environment. Do not assume Bash syntax inside tmux panes.
105 +Before creating or driving panes, identify the user's shell and terminal
107 +Do not assume Bash syntax inside tmux panes.
77 -- Nushell: use `;`, not `&&`; env vars use `$env.NAME`; avoid Bash snippets in the pane. **Load the `nushell` skill** for exact syntax.
78 -- Fish: use `;` or `and`, not Bash-only constructs.
79 -- PowerShell: use `;`; env vars use `$env:NAME`.
80 -- Bash/Zsh/Sh: `&&`, `VAR=value cmd`, and POSIX snippets are OK.
81 -- If unsure, run complex setup from the agent's `bash` tool, not by pasting Bash into the user's pane.
82 -- When sending commands to tmux panes, prefer simple one-command lines or shell-specific forms.
120 + use `;`, not `&&`; env vars use `$env.NAME`; avoid Bash snippets in the pane.
121 + **Load the `nushell` skill** for exact syntax.
123 + use `;` or `and`, not Bash-only constructs.
125 + use `;`; env vars use `$env:NAME`.
127 + `&&`, `VAR=value cmd`, and POSIX snippets are OK.
128 +- If unsure, run complex setup from the agent's `bash` tool, not by pasting Bash
129 + into the user's pane.
130 +- When sending commands to tmux panes, prefer simple one-command lines or
131 + shell-specific forms.
84 133 ##
Observable Terminal + tmux Setup 86 135 Load the **`tmux`** skill before writing any tmux orchestration.
88 -Create a dedicated tmux socket so the test environment has exactly one session/window without touching the user's existing tmux:
137 +Create a dedicated tmux socket so the test environment has exactly one
138 +session/window without touching the user's existing tmux:
91 141 RUN_ID
="pi-ext-$(date +%Y%m%d-%H%M%S)" 125 175 tmux -L "$SOCKET" list-windows -t "$SESSION"
128 -Use a shell-aware pane launch command. **Keep it simple** — one command per `send-keys`, no shell operators inside the pane if Nushell is running:
178 +Use a shell-aware pane launch command.
179 +**Keep it simple** — one command per `send-keys`, no shell operators inside the
180 +pane if Nushell is running:
131 183 PI_CMD="pi --no-extensions --no-skills --no-prompt-templates --no-context-files -e ./extensions/<target>/index.ts"
153 -- `tmux -L "$SOCKET" list-sessions` shows exactly one session: `live`
154 -- `tmux -L "$SOCKET" list-windows -t live` shows exactly one window: `main`
205 +- `tmux -L "$SOCKET" list-sessions` shows exactly one session:
207 +- `tmux -L "$SOCKET" list-windows -t live` shows exactly one window:
155 209 - more panes are OK; more sessions/windows are not
157 -Open/attach that tmux session in the current terminal emulator, preferring an actual split over a new OS window:
211 +Open/attach that tmux session in the current terminal emulator, preferring an
212 +actual split over a new OS window:
164 219 wezterm cli split-pane --right --cwd "$ROOT" -- tmux -L "$SOCKET" attach -t "$SESSION"
167 -2. Ghostty / Kitty / Alacritty / GNOME Terminal / Terminal.app / Windows Terminal / other popular terminals:
222 +1. Ghostty / Kitty / Alacritty / GNOME Terminal / Terminal.app / Windows
223 + Terminal / other popular terminals:
169 -- Detect via env and available commands: `WEZTERM_PANE`, `GHOSTTY_*`, `KITTY_WINDOW_ID`, `ALACRITTY_*`, `TERM_PROGRAM`, `WT_SESSION`, `TMUX`, `uname`.
225 +- Detect via env and available commands:
226 + `WEZTERM_PANE`, `GHOSTTY_*`, `KITTY_WINDOW_ID`, `ALACRITTY_*`, `TERM_PROGRAM`,
227 + `WT_SESSION`, `TMUX`, `uname`.
170 228 - Use the terminal's CLI to open a split/window attached to:
173 231 tmux -L "$SOCKET" attach -t "$SESSION"
176 -3. If no terminal split/window can be controlled safely:
234 +1. If no terminal split/window can be controlled safely:
178 236 - Keep the tmux server running.
179 237 - Print the exact attach command prominently.
192 250 Adjust only when needed:
194 -- support extension: add another `-e ./extensions/<support>/index.ts`
195 -- local package extension path: use exact path
196 -- scenario requires skills/prompts/context: remove only the needed isolation flag
197 -- scenario requires previous session behavior: use normal session mode intentionally and note why
252 +- support extension:
253 + add another `-e ./extensions/<support>/index.ts`
254 +- local package extension path:
256 +- scenario requires skills/prompts/context:
257 + remove only the needed isolation flag
258 +- scenario requires previous session behavior:
259 + use normal session mode intentionally and note why
199 -Do not force a model unless scenario requires model-specific behavior. Default configured model is preferred. For very long validation, it is OK to perform one secondary run with another available real model if useful.
261 +Do not force a model unless scenario requires model-specific behavior.
262 +Default configured model is preferred.
263 +For very long validation, it is OK to perform one secondary run with another
264 +available real model if useful.
201 266 ##
Critical Verification Rules 203 -The Pi agent inside tmux is the system under test. Treat it like an unreliable junior engineer trying to impress you.
268 +The Pi agent inside tmux is the system under test.
269 +Treat it like an unreliable junior engineer trying to impress you.
205 -- Do not trust its claims of success, failure, verification, implementation, or UX quality.
271 +- Do not trust its claims of success, failure, verification, implementation, or
206 273 - Independently inspect files it says it changed.
207 274 - Independently rerun commands it says passed.
208 275 - Independently inspect browser/terminal screenshots for UI claims.
209 -- **Model/provider gotcha**: when an extension frontmatter says `model: kimi/kimi-for-coding`, verify the exact provider name with `pi --list-models` and with `getModel(provider, id)` from `@earendil-works/pi-ai`. Invalid provider strings often return `undefined`, and `createAgentSession` may silently fall back to the parent session's default model.
210 -- Check whether it ignored instructions, skipped workflows, invented evidence, hid errors, or reframed the task smaller.
276 +- **Model/provider gotcha**:
277 + when an extension frontmatter says `model:
278 + kimi/kimi-for-coding`, verify the exact provider name with `pi --list-models`
279 + and with `getModel(provider, id)` from `@earendil-works/pi-ai`.
280 + Invalid provider strings often return `undefined`, and `createAgentSession`
281 + may silently fall back to the parent session's default model.
282 +- Check whether it ignored instructions, skipped workflows, invented evidence,
283 + hid errors, or reframed the task smaller.
211 284 - Record every unsupported claim as a finding until verified.
212 -- Prefer hard evidence: file diffs, command output, screenshots, browser console/network logs, session JSONL, pane captures, **and direct SDK integration tests** when the TUI is not observable.
213 -- Be concise but harsh in notes. No praise unless evidence earned it.
285 +- Prefer hard evidence:
286 + file diffs, command output, screenshots, browser console/network logs, session
287 + JSONL, pane captures, **and direct SDK integration tests** when the TUI is not
289 +- Be concise but harsh in notes.
290 + No praise unless evidence earned it.
215 292 ##
Human-like Driving Rules 219 296 - type commands/prompts into pane 1
220 297 - wait for UI states to settle
221 298 - read terminal output
222 -- **TUI gotcha**: Pi uses the alternate screen buffer. `tmux capture-pane` often shows only the status bar, not chat content. If you cannot see responses, verify via file-system artifacts, session JSONL, or direct SDK tests instead of fighting the TUI.
223 -- **Key concatenation gotcha**: `C-c C-u` does not reliably clear Pi's input widget. Always send `C-c`, wait, then `C-u`, wait, then type the new command. If you see merged commands like `/foo/bar`, the previous input was not cleared.
224 -- use extension slash commands, shortcuts, reloads, session switches, browser UIs, or tools only as real users would
225 -- if browser UI opens, inspect it and record UX findings with browser automation, not just terminal logs
226 -- test happy path, rough path, cancel/abort, reload/session lifecycle, browser lifecycle, and one realistic long-running path
227 -- observe latency and visual stability; note jank, flicker, unclear status, noisy widgets, bad footer/status behavior, stale state, or confusing errors
300 + Pi uses the alternate screen buffer.
301 + `tmux capture-pane` often shows only the status bar, not chat content.
302 + If you cannot see responses, verify via file-system artifacts, session JSONL,
303 + or direct SDK tests instead of fighting the TUI.
304 +- **Key concatenation gotcha**:
305 + `C-c C-u` does not reliably clear Pi's input widget.
306 + Always send `C-c`, wait, then `C-u`, wait, then type the new command.
307 + If you see merged commands like `/foo/bar`, the previous input was not
309 +- use extension slash commands, shortcuts, reloads, session switches, browser
310 + UIs, or tools only as real users would
311 +- if browser UI opens, inspect it and record UX findings with browser
312 + automation, not just terminal logs
313 +- test happy path, rough path, cancel/abort, reload/session lifecycle, browser
314 + lifecycle, and one realistic long-running path
315 +- observe latency and visual stability; note jank, flicker, unclear status,
316 + noisy widgets, bad footer/status behavior, stale state, or confusing errors
228 317 - avoid unit-test shortcuts unless using them only to understand implementation
230 -This is expected to take time. Be thorough.
319 +This is expected to take time.
232 322 ##
Browser UI Testing 234 -Some Pi extensions open local HTML pages. If a page opens or a URL is printed, that page is part of the extension UX and must be tested.
324 +Some Pi extensions open local HTML pages.
325 +If a page opens or a URL is printed, that page is part of the extension UX and
236 328 Browser protocol:
238 -1. Detect URL/browser from terminal output, server info files, `xdg-open`/`open` behavior, or extension docs.
239 -2. If Chrome/Chromium is open or appropriate, load the `chrome-cdp` skill and use it to inspect/drive the live page.
240 -3. If Firefox is open or appropriate, load the `firefox-bidi` skill and use it to inspect/drive the live page; launching a managed Firefox is OK when needed.
241 -4. Interact like a human: click controls, type fields, resize if relevant, reload, close/reopen, test stale sessions, and verify events flow back to Pi.
242 -5. Inspect console errors, failed network requests, DOM state, accessibility labels where practical, and visual layout.
243 -6. Record browser findings in the notes with URL, browser, steps, expected/actual, and evidence.
330 +1. Detect URL/browser from terminal output, server info files, `xdg-open`/`open`
331 + behavior, or extension docs.
332 +2. If Chrome/Chromium is open or appropriate, load the `chrome-cdp` skill and
333 + use it to inspect/drive the live page.
334 +3. If Firefox is open or appropriate, load the `firefox-bidi` skill and use it
335 + to inspect/drive the live page; launching a managed Firefox is OK when
337 +4. Interact like a human:
338 + click controls, type fields, resize if relevant, reload, close/reopen, test
339 + stale sessions, and verify events flow back to Pi.
340 +5. Inspect console errors, failed network requests, DOM state, accessibility
341 + labels where practical, and visual layout.
342 +6. Record browser findings in the notes with URL, browser, steps,
343 + expected/actual, and evidence.
245 -Do not skip browser testing because terminal behavior passed. If browser automation is unavailable, mark the browser portion blocked and say exactly what is missing.
345 +Do not skip browser testing because terminal behavior passed.
346 +If browser automation is unavailable, mark the browser portion blocked and say
347 +exactly what is missing.
247 349 ##
Notes Discipline 270 372 - terminal emulator detected
271 373 - user shell and tmux pane shell detected
272 374 - tmux socket/session/window
273 -- browser used, URL tested, console/network issues, and browser skill used when applicable
375 +- browser used, URL tested, console/network issues, and browser skill used when
274 377 - screenshots/GIF paths only if created
275 378 - every bug/UX issue with reproduction steps
276 379 - non-bugs worth preserving
282 385 1. Do not kill the tmux server automatically.
283 386 2. Leave tmux running for user inspection.
284 387 3. Give attach and kill commands.
285 -4. Ask: `Do you want me to close the tmux session now?`
389 + `Do you want me to close the tmux session now?`
286 390 5. Only run `tmux -L <socket> kill-server` after the user says yes.
287 391 6. Ensure notes file has final summary.
301 405 - [severity] title — evidence/repro
304 -If blocked by terminal control, auth/model config, missing extension path, or unsafe destructive scenario, stop and ask for the smallest missing input. If testing completes successfully, still leave tmux alive and ask whether to close it.
408 +If blocked by terminal control, auth/model config, missing extension path, or
409 +unsafe destructive scenario, stop and ask for the smallest missing input.
410 +If testing completes successfully, still leave tmux alive and ask whether to
306 413 ## Fallback When TUI Driving Fails
308 -If `tmux capture-pane` cannot read Pi's chat output (alternate-screen-buffer issue), do not abandon the test. Fall back in this order:
415 +If `tmux capture-pane` cannot read Pi's chat output (alternate-screen-buffer
416 +issue), do not abandon the test.
417 +Fall back in this order:
310 -1. **File-system artifacts** — check extension state dirs, session JSONL, or any files the extension writes.
311 -2. **Direct SDK integration tests** — import the extension's code, call its internal helpers, or create real `createAgentSession` calls with real models to exercise the same code paths without the TUI.
312 -3. **Process inspection** — `ls -l /proc/<pid>/fd/` and `ps` to confirm the process is alive and which files it holds open.
313 -4. **Session file forensics** — read Pi's JSONL session files with `jq` to reconstruct what happened.
419 +1. **File-system artifacts** — check extension state dirs, session JSONL, or any
420 + files the extension writes.
421 +2. **Direct SDK integration tests** — import the extension's code, call its
422 + internal helpers, or create real `createAgentSession` calls with real models
423 + to exercise the same code paths without the TUI.
424 +3. **Process inspection** — `ls -l /proc/<pid>/fd/` and `ps` to confirm the
425 + process is alive and which files it holds open.
426 +4. **Session file forensics** — read Pi's JSONL session files with `jq` to
427 + reconstruct what happened.
315 -These fallbacks are not "cheating" — they are the harness engineer doing their job when the terminal layer is opaque.
429 +These fallbacks are not "cheating" — they are the harness engineer doing their
430 +job when the terminal layer is opaque.