repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

chore: add model test prompts and fix tldr tests

d9ef6f353fc5b67444f2b1d255cb96e22fa183a8

Oliver Jan Krylow <oliver@bugabinga.net> ·

unsigned

Browse filesCompare with HEAD

Changed files (8)

.pi/prompts/live-extension-test.md

Modified
30 30 - Keep running notes in a temp markdown file while testing.
31 31 - Finish with a concise findings report and path to the notes file.
33 +## Coding-Harness Engineer Mandate
35 +You are not merely a user of Pi — you are a **coding-harness engineer**. Your job is to actively engineer the harness around Pi so the extension under test performs at its best. This means:
37 +- **Seek the gold hand.** Before launching Pi, think: what would the ideal test environment look like for this extension? Then build it.
38 +- **Use project-local Pi modifications.** Scan the repo for local skills (`./skills/`), local prompts (`./.pi/prompts/`), local context files (`./.pi/context-files/`), local extensions (`./extensions/`), and any workspace-level `.pi/` config. If a temporary local skill, prompt template, context file, or small scaffolding extension would improve the test, create it in the project's `.pi/` tree (or appropriate local path). Clean it up after testing unless it represents a genuine improvement worth keeping.
39 +- **Load global extensions if they provide testing infrastructure.** You may load global extensions (e.g., `chrome-cdp`, `firefox-bidi`, `tmux`, `research`) when they give you better observability, setup, or control. **Never modify any global extension except the one under test.**
40 +- **Tweak the harness, not the goal.** Every modification you make — env vars, preloaded skills, scaffolded test projects, injected prompts, context files — must serve the single purpose of improving the extension under test. Do not drift into generic harness polishing.
41 +- **Remember your true north.** Even though you are pretending to be a coding-harness engineer, the ultimate goal is to improve the extension under test. If a harness change reveals a bug, missing feature, or UX flaw in the extension, file it as a finding and, when appropriate, patch the extension itself.
33 43 ## Parse Scenario
35 45 Extract from `$ARGUMENTS`:
44 54 If target is ambiguous, inspect repo structure and choose the most likely extension path. If still ambiguous, ask one focused question.
56 +## Skills to Load
58 +Load these skills when their domain is active:
59 +- **`tmux`** — for all session/pane creation, `send-keys`, `capture-pane`, and multi-pane orchestration.
60 +- **`nushell`** — when the pane shell is `nu` or `nu.exe`. Nushell has no `&&`, no `$VAR` interpolation, and `;` is the statement separator.
61 +- **`wezterm`** — when `WEZTERM_PANE` is set and you need to split panes or attach via WezTerm CLI.
46 63 ## Shell Probe
48 65 Before creating or driving panes, identify the user's shell and terminal environment. Do not assume Bash syntax inside tmux panes.
58 75 Shell rules:
60 -- Nushell: use `;`, not `&&`; env vars use `$env.NAME`; avoid Bash snippets in the pane.
77 +- Nushell: use `;`, not `&&`; env vars use `$env.NAME`; avoid Bash snippets in the pane. **Load the `nushell` skill** for exact syntax.
61 78 - Fish: use `;` or `and`, not Bash-only constructs.
62 79 - PowerShell: use `;`; env vars use `$env:NAME`.
63 80 - Bash/Zsh/Sh: `&&`, `VAR=value cmd`, and POSIX snippets are OK.
67 84 ## Observable Terminal + tmux Setup
86 +Load the **`tmux`** skill before writing any tmux orchestration.
69 88 Create a dedicated tmux socket so the test environment has exactly one session/window without touching the user's existing tmux:
71 90 ```sh
106 125 tmux -L "$SOCKET" list-windows -t "$SESSION"
109 -Use a shell-aware pane launch command:
128 +Use a shell-aware pane launch command. **Keep it simple** — one command per `send-keys`, no shell operators inside the pane if Nushell is running:
112 131 PI_CMD="pi --no-extensions --no-skills --no-prompt-templates --no-context-files -e ./extensions/<target>/index.ts"
113 132 case "$PANE_SHELL" in
114 - nu|nu.exe|fish|pwsh|powershell.exe) PANE_LAUNCH="cd '$ROOT'; $PI_CMD" ;;
115 - *) PANE_LAUNCH="cd '$ROOT' && $PI_CMD" ;;
133 + nu|nu.exe)
134 + # Nushell: ';' is the statement separator. '&&' does not exist.
135 + tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
136 + ;;
137 + fish)
138 + # Fish: ';' works, 'and' is the keyword form.
139 + tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
140 + ;;
141 + pwsh|powershell.exe)
142 + # PowerShell: ';' is the statement separator.
143 + tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT'; $PI_CMD" C-m
144 + ;;
145 + *)
146 + tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "cd '$ROOT' && $PI_CMD" C-m
147 + ;;
118 -tmux -L "$SOCKET" send-keys -t "$SESSION:main.1" "$PANE_LAUNCH" C-m
121 151 Invariant:
129 159 1. WezTerm:
161 +Load the **`wezterm`** skill before using WezTerm CLI commands.
132 164 wezterm cli split-pane --right --cwd "$ROOT" -- tmux -L "$SOCKET" attach -t "$SESSION"
174 206 - Independently inspect files it says it changed.
175 207 - Independently rerun commands it says passed.
176 208 - Independently inspect browser/terminal screenshots for UI claims.
209 +- **Model/provider gotcha**: when an extension frontmatter says `model: kimi/kimi-for-coding`, verify the exact provider name with `pi --list-models` and with `getModel(provider, id)` from `@earendil-works/pi-ai`. Invalid provider strings often return `undefined`, and `createAgentSession` may silently fall back to the parent session's default model.
177 210 - Check whether it ignored instructions, skipped workflows, invented evidence, hid errors, or reframed the task smaller.
178 211 - Record every unsupported claim as a finding until verified.
179 -- Prefer hard evidence: file diffs, command output, screenshots, browser console/network logs, session JSONL, pane captures.
212 +- Prefer hard evidence: file diffs, command output, screenshots, browser console/network logs, session JSONL, pane captures, **and direct SDK integration tests** when the TUI is not observable.
180 213 - Be concise but harsh in notes. No praise unless evidence earned it.
182 215 ## Human-like Driving Rules
186 219 - type commands/prompts into pane 1
187 220 - wait for UI states to settle
188 221 - read terminal output
222 +- **TUI gotcha**: Pi uses the alternate screen buffer. `tmux capture-pane` often shows only the status bar, not chat content. If you cannot see responses, verify via file-system artifacts, session JSONL, or direct SDK tests instead of fighting the TUI.
223 +- **Key concatenation gotcha**: `C-c C-u` does not reliably clear Pi's input widget. Always send `C-c`, wait, then `C-u`, wait, then type the new command. If you see merged commands like `/foo/bar`, the previous input was not cleared.
189 224 - use extension slash commands, shortcuts, reloads, session switches, browser UIs, or tools only as real users would
190 225 - if browser UI opens, inspect it and record UX findings with browser automation, not just terminal logs
191 226 - test happy path, rough path, cancel/abort, reload/session lifecycle, browser lifecycle, and one realistic long-running path
269 304 If blocked by terminal control, auth/model config, missing extension path, or unsafe destructive scenario, stop and ask for the smallest missing input. If testing completes successfully, still leave tmux alive and ask whether to close it.
306 +## Fallback When TUI Driving Fails
308 +If `tmux capture-pane` cannot read Pi's chat output (alternate-screen-buffer issue), do not abandon the test. Fall back in this order:
310 +1. **File-system artifacts** — check extension state dirs, session JSONL, or any files the extension writes.
311 +2. **Direct SDK integration tests** — import the extension's code, call its internal helpers, or create real `createAgentSession` calls with real models to exercise the same code paths without the TUI.
312 +3. **Process inspection** — `ls -l /proc/<pid>/fd/` and `ps` to confirm the process is alive and which files it holds open.
313 +4. **Session file forensics** — read Pi's JSONL session files with `jq` to reconstruct what happened.
315 +These fallbacks are not "cheating" — they are the harness engineer doing their job when the terminal layer is opaque.

.pi/prompts/test-kimi-broken.md

Added
new file mode 100644
1 +---
2 +description: Test broken kimi alias
3 +tools: [read, bash]
4 +model: kimi/kimi-for-coding
5 +accent: error
6 +---
7 +Say "hello from broken kimi" and then read the file /tmp/pi-ext-20260529-213419-findings.md and report its first line.

.pi/prompts/test-kimi.md

Added
new file mode 100644
1 +---
2 +description: Test kimi provider
3 +tools: [read, bash]
4 +model: kimi-coding/kimi-for-coding
5 +accent: syntaxKeyword
6 +---
7 +Say "hello from kimi" and then read the file /tmp/pi-ext-20260529-213419-findings.md and report its first line.

.pi/prompts/test-minimax.md

Added
new file mode 100644
1 +---
2 +description: Test minimax provider
3 +tools: [read, bash]
4 +model: minimax/MiniMax-M2.7
5 +accent: success
6 +---
7 +Say "hello from minimax" and then read the file /tmp/pi-ext-20260529-213419-findings.md and report its first line.

.pi/prompts/test-openai.md

Added
new file mode 100644
1 +---
2 +description: Test openai provider
3 +tools: [read, bash]
4 +model: openai/gpt-5.2
5 +accent: warning
6 +---
7 +Say "hello from openai" and then read the file /tmp/pi-ext-20260529-213419-findings.md and report its first line.

.pi/prompts/test-zai.md

Added
new file mode 100644
1 +---
2 +description: Test zai provider
3 +tools: [read, bash]
4 +model: zai/glm-5.1
5 +accent: syntaxFunction
6 +---
7 +Say "hello from zai" and then read the file /tmp/pi-ext-20260529-213419-findings.md and report its first line.

extensions/tldr/__tests__/tldr.test.ts

Modified
209 209 maxRetries: 0,
210 210 cacheRetention: "none",
212 - expect(ctx.widgets["tldr"]).toContain("│ Reading package metadata. │");
212 + const factory = ctx.widgets["tldr"] as (tui: unknown, theme: unknown) => { render(width: number): string[] };
213 + expect(typeof factory).toBe("function");
214 + const component = factory({}, {});
215 + expect(component.render(80)[1]).toContain("Reading package metadata.");
215 218 it("coalesces normal activity until the scheduler flushes", async () => {
234 237 scheduler.flush();
235 238 await settle();
236 239 expect(calls).toBe(1);
237 - expect(ctx.widgets["tldr"]).toContain("│ Update 1. │");
240 + const factory = ctx.widgets["tldr"] as (tui: unknown, theme: unknown) => { render(width: number): string[] };
241 + const component = factory({}, {});
242 + expect(component.render(80)[1]).toContain("Update 1.");
240 245 it("drops stale model results after a fresh run starts", async () => {

extensions/tldr/index.ts

Modified
183 183 return lines.length ? lines : [""];
186 -function boxLines(text: string): string[] {
186 +function boxLines(text: string, terminalWidth = 80): string[] {
187 187 const safe = sanitizeTldrText(text);
188 188 if (!safe) return [];
189 189 const title = " tldr ";
190 - const contentWidth = Math.max(10, Math.min(76, Array.from(safe).length));
191 - const width = contentWidth + 4;
190 + const boxWidth = Math.max(10, terminalWidth);
191 + const contentWidth = boxWidth - 4;
192 192 const contentLines = wrapText(safe, contentWidth).map((line) => {
193 193 const lineLength = Array.from(line).length;
194 194 return line + " ".repeat(Math.max(0, contentWidth - lineLength));
196 196 return [
197 - `╭${title}${"─".repeat(Math.max(1, width - Array.from(title).length - 2))}╮`,
197 + `╭${title}${"─".repeat(Math.max(1, boxWidth - Array.from(title).length - 2))}╮`,
198 198 ...contentLines.map((line) => `│ ${line} │`),
199 - `╰${"─".repeat(width - 2)}╯`,
199 + `╰${"─".repeat(boxWidth - 2)}╯`,
204 204 if (!ctx.hasUI) return;
205 205 const safe = sanitizeTldrText(text);
206 206 if (!safe) return clearTldr(ctx);
207 - ctx.ui.setWidget(WIDGET_KEY, boxLines(safe));
207 + ctx.ui.setWidget(WIDGET_KEY, (_tui, _theme) => ({
208 + render(width: number) {
209 + return boxLines(safe, width);
210 + },
211 + }));
210 214 function clearTldr(ctx: ExtensionContext): void {