30 30 - Keep running notes in a temp markdown file while testing.
31 31 - Finish with a concise findings report and path to the notes file.
33 +##
Coding-Harness Engineer Mandate 35 +You are not merely a user of Pi — you are a **coding-harness engineer**. Your job is to actively engineer the harness around Pi so the extension under test performs at its best. This means:
37 +- **Seek the gold hand.** Before launching Pi, think: what would the ideal test environment look like for this extension? Then build it.
38 +- **Use project-local Pi modifications.** Scan the repo for local skills (`./skills/`), local prompts (`./.pi/prompts/`), local context files (`./.pi/context-files/`), local extensions (`./extensions/`), and any workspace-level `.pi/` config. If a temporary local skill, prompt template, context file, or small scaffolding extension would improve the test, create it in the project's `.pi/` tree (or appropriate local path). Clean it up after testing unless it represents a genuine improvement worth keeping.
39 +- **Load global extensions if they provide testing infrastructure.** You may load global extensions (e.g., `chrome-cdp`, `firefox-bidi`, `tmux`, `research`) when they give you better observability, setup, or control. **Never modify any global extension except the one under test.**
40 +- **Tweak the harness, not the goal.** Every modification you make — env vars, preloaded skills, scaffolded test projects, injected prompts, context files — must serve the single purpose of improving the extension under test. Do not drift into generic harness polishing.
41 +- **Remember your true north.** Even though you are pretending to be a coding-harness engineer, the ultimate goal is to improve the extension under test. If a harness change reveals a bug, missing feature, or UX flaw in the extension, file it as a finding and, when appropriate, patch the extension itself.
35 45 Extract from `$ARGUMENTS`:
44 54 If target is ambiguous, inspect repo structure and choose the most likely extension path. If still ambiguous, ask one focused question.
58 +Load these skills when their domain is active:
59 +- **`tmux`** — for all session/pane creation, `send-keys`, `capture-pane`, and multi-pane orchestration.
60 +- **`nushell`** — when the pane shell is `nu` or `nu.exe`. Nushell has no `&&`, no `$VAR` interpolation, and `;` is the statement separator.
61 +- **`wezterm`** — when `WEZTERM_PANE` is set and you need to split panes or attach via WezTerm CLI.
48 65 Before creating or driving panes, identify the user's shell and terminal environment. Do not assume Bash syntax inside tmux panes.
60 -- Nushell: use `;`, not `&&`; env vars use `$env.NAME`; avoid Bash snippets in the pane.
77 +- Nushell: use `;`, not `&&`; env vars use `$env.NAME`; avoid Bash snippets in the pane. **Load the `nushell` skill** for exact syntax.
61 78 - Fish: use `;` or `and`, not Bash-only constructs.
62 79 - PowerShell: use `;`; env vars use `$env:NAME`.
63 80 - Bash/Zsh/Sh: `&&`, `VAR=value cmd`, and POSIX snippets are OK.
67 84 ##
Observable Terminal + tmux Setup 86 +Load the **`tmux`** skill before writing any tmux orchestration.
69 88 Create a dedicated tmux socket so the test environment has exactly one session/window without touching the user's existing tmux:
106 125 tmux -L "$SOCKET" list-windows -t "$SESSION"
109 -Use a shell-aware pane launch command:
128 +Use a shell-aware pane launch command. **Keep it simple** — one command per `send-keys`, no shell operators inside the pane if Nushell is running:
112 131 PI_CMD="pi --no-extensions --no-skills --no-prompt-templates --no-context-files -e ./extensions/<target>/index.ts"
113 132 case "$PANE_SHELL" in
114 - nu|nu.exe|fish|pwsh|powershell.exe) PANE_LAUNCH="cd '$ROOT'; $PI_CMD" ;;
115 - *) PANE_LAUNCH="cd '$ROOT' && $PI_CMD" ;;
134 + # Nushell: ';' is the statement separator. '&&' does not exist.
135 + tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
138 + # Fish: ';' works, 'and' is the keyword form.
139 + tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
141 + pwsh|powershell.exe)
142 + # PowerShell: ';' is the statement separator.
143 + tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
146 + tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT' && $PI_CMD" C-m
118 -tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "$PANE_LAUNCH" C-m
161 +Load the **`wezterm`** skill before using WezTerm CLI commands.
132 164 wezterm cli split-pane --right --cwd
"$ROOT" -- tmux -L
"$SOCKET" attach -t
"$SESSION" 174 206 - Independently inspect files it says it changed.
175 207 - Independently rerun commands it says passed.
176 208 - Independently inspect browser/terminal screenshots for UI claims.
209 +- **Model/provider gotcha**: when an extension frontmatter says `model: kimi/kimi-for-coding`, verify the exact provider name with `pi --list-models` and with `getModel(provider, id)` from `@earendil-works/pi-ai`. Invalid provider strings often return `undefined`, and `createAgentSession` may silently fall back to the parent session's default model.
177 210 - Check whether it ignored instructions, skipped workflows, invented evidence, hid errors, or reframed the task smaller.
178 211 - Record every unsupported claim as a finding until verified.
179 -- Prefer hard evidence: file diffs, command output, screenshots, browser console/network logs, session JSONL, pane captures.
212 +- Prefer hard evidence: file diffs, command output, screenshots, browser console/network logs, session JSONL, pane captures, **and direct SDK integration tests** when the TUI is not observable.
180 213 - Be concise but harsh in notes. No praise unless evidence earned it.
182 215 ##
Human-like Driving Rules 186 219 - type commands/prompts into pane 1
187 220 - wait for UI states to settle
188 221 - read terminal output
222 +- **TUI gotcha**: Pi uses the alternate screen buffer. `tmux capture-pane` often shows only the status bar, not chat content. If you cannot see responses, verify via file-system artifacts, session JSONL, or direct SDK tests instead of fighting the TUI.
223 +- **Key concatenation gotcha**: `C-c C-u` does not reliably clear Pi's input widget. Always send `C-c`, wait, then `C-u`, wait, then type the new command. If you see merged commands like `/foo/bar`, the previous input was not cleared.
189 224 - use extension slash commands, shortcuts, reloads, session switches, browser UIs, or tools only as real users would
190 225 - if browser UI opens, inspect it and record UX findings with browser automation, not just terminal logs
191 226 - test happy path, rough path, cancel/abort, reload/session lifecycle, browser lifecycle, and one realistic long-running path
269 304 If blocked by terminal control, auth/model config, missing extension path, or unsafe destructive scenario, stop and ask for the smallest missing input. If testing completes successfully, still leave tmux alive and ask whether to close it.
306 +## Fallback When TUI Driving Fails
308 +If `tmux capture-pane` cannot read Pi's chat output (alternate-screen-buffer issue), do not abandon the test. Fall back in this order:
310 +1. **File-system artifacts** — check extension state dirs, session JSONL, or any files the extension writes.
311 +2. **Direct SDK integration tests** — import the extension's code, call its internal helpers, or create real `createAgentSession` calls with real models to exercise the same code paths without the TUI.
312 +3. **Process inspection** — `ls -l /proc/<pid>/fd/` and `ps` to confirm the process is alive and which files it holds open.
313 +4. **Session file forensics** — read Pi's JSONL session files with `jq` to reconstruct what happened.
315 +These fallbacks are not "cheating" — they are the harness engineer doing their job when the terminal layer is opaque.