repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

docs(tune): plan cross-platform runner

f5efdebfccd9354b6711b7174f594f1afc08a4c4

Oliver Krylow <o.krylow@isp-insoft.de> ·

unsigned

Browse filesCompare with HEAD

Changed files (2)

docs/super/plans/2026-07-16-tune-cross-platform-runner.md

Added
new file mode 100644
1 +# Tune Cross-Platform Runner Implementation Plan
3 +> **For agentic workers:** REQUIRED SUB-SKILL: Use /skill:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
5 +**Goal:** Remove Tune's Bash/WSL dependency by running its fixed MJS benchmark with Pi's Node executable and invoking Git directly without regressing experiment behavior.
7 +**Architecture:** Keep production orchestration in `extensions/tune/index.ts`. Replace only process launch, termination, tool schema, and Git discard; use `process.execPath` for every Tune MJS file and preserve existing state, output, metrics, checks, VCS, and UI behavior.
9 +**Tech Stack:** TypeScript, Node standard library, Pi extension API, TypeBox, Vitest, Pi test harness, Mise
11 +**Roadmap:** None
13 +**Phase:** Single-plan implementation
15 +---
17 +### Task 1: Establish Regression Harness and No-Bash RED
19 +**Files:**
21 +- Modify: `extensions/tune/__tests__/harness.test.ts`
22 +- Modify: `scripts/test_extensions_static_check.mjs`
24 +- [ ] **Step 1: Add a deterministic active-session helper**
26 +Activate Tune without invoking `/tune`, because `/tune` sends an asynchronous user message.
27 +Create the persisted config, acquire the real session lock, and reconstruct through `session_tree`:
29 +```ts
30 +async function activeTune(maxIterations: number | null = null) {
31 + const extensionPath = path.resolve(import.meta.dirname, "../index.ts");
32 + t = await createTestSession({ extensions: [extensionPath] });
33 + fs.mkdirSync(path.join(t.cwd, ".pi", "tune"), { recursive: true });
34 + fs.writeFileSync(
35 + path.join(t.cwd, ".pi", "tune", "session.jsonl"),
36 + "{\"type\":\"config\",\"name\":\"runtime\",\"metricName\":\"score\",\"metricUnit\":\"\",\"bestDirection\":\"lower\"}\n",
37 + );
38 + if (maxIterations !== null) {
39 + fs.writeFileSync(
40 + path.join(t.cwd, ".pi", "settings.json"),
41 + JSON.stringify({ tune: { maxIterations } }),
42 + );
43 + }
44 + __test.acquireTuneLock(t.cwd, t.session.sessionManager.getSessionId());
45 + await t.session.extensionRunner.emit({
46 + type: "session_tree",
47 + newLeafId: null,
48 + oldLeafId: null,
49 + });
50 + return t.session.extensionRunner.extensions[0];
52 +```
54 +Import `calls`, `says`, and `when` from the harness.
55 +All public benchmark and VCS assertions must call the registered tools through `t.run`.
56 +Direct definition execution is allowed only where the harness DSL cannot expose `onUpdate` or an external `AbortSignal`.
58 +- [ ] **Step 2: Add the static no-Bash contract**
60 +Add before production changes:
62 +```js
63 +const tuneIndex = readFileSync(join(ROOT, "extensions/tune/index.ts"), "utf8");
64 +assert(!tuneIndex.includes("spawn(\"bash\""));
65 +assert(!tuneIndex.includes("exec(\"bash\""));
66 +```
68 +- [ ] **Step 3: Verify RED**
70 +Run:
72 +```text
73 +mise run check
74 +```
76 +Expected: static checking fails because Tune still contains both Bash invocations.
78 +### Task 2: Replace Bash-Based Git Discard
80 +**Files:**
82 +- Modify: `extensions/tune/__tests__/unit.test.ts`
83 +- Modify: `extensions/tune/__tests__/harness.test.ts`
84 +- Modify: `extensions/tune/index.ts:715-747`
86 +- [ ] **Step 1: Add exact-call and failure-order tests**
88 +Add mocked unit cases for direct arguments and failure ordering:
90 +```ts
91 +it("uses direct Git calls on discard", async () => {
92 + const calls: Array<[string, string[]]> = [];
93 + const exec = async (cmd: string, args: string[]) => {
94 + calls.push([cmd, args]);
95 + return { code: cmd === "jj" ? 1 : 0, stdout: "", stderr: "" };
96 + };
97 + await __test.discardVcsChanges(exec, "/repo", "discard");
98 + expect(calls).toEqual([
99 + ["jj", ["root"]],
100 + ["git", [
101 + "checkout",
102 + "--",
103 + ".",
104 + ":(exclude,glob).pi/tune/**",
105 + ":(exclude,glob).pi/tune/*",
106 + ]],
107 + ["git", ["clean", "-fd", "-e", ".pi/tune", "-e", ".pi/tune/**"]],
108 + ]);
109 +});
111 +it("does not clean after checkout failure", async () => {
112 + const calls: string[] = [];
113 + const exec = async (cmd: string, args: string[]) => {
114 + calls.push(`${cmd} ${args[0]}`);
115 + if (cmd === "jj") return { code: 1, stdout: "", stderr: "" };
116 + return { code: 1, stdout: "", stderr: "checkout failed" };
117 + };
118 + await expect(__test.discardVcsChanges(exec, "/repo", "discard"))
119 + .rejects.toThrow("git checkout failed: checkout failed");
120 + expect(calls).toEqual(["jj root", "git checkout"]);
121 +});
122 +```
124 +Add the equivalent nonzero `git clean` case and require `git clean failed:` with bounded stderr.
126 +- [ ] **Step 2: Add a real Git repository test through `log_experiment`**
128 +In `harness.test.ts`, initialize a real temporary Git repository, configure local author identity, commit `tracked.txt`, then modify it and add `untracked.txt` plus `.pi/tune/ideas.md`.
129 +Activate Tune with Task 1's helper and call:
131 +```ts
132 +await t!.run(
133 + when("discard regression", [
134 + calls("log_experiment", {
135 + commit: "0000000",
136 + metric: 100,
137 + status: "discard",
138 + description: "regression",
139 + asi: { hypothesis: "worse", next_action_hint: "revert" },
140 + }),
141 + says("discarded"),
142 + ]),
144 +```
146 +Assert the tracked file is restored, the untracked file is deleted, and both `ideas.md` and the appended discard entry in `session.jsonl` survive.
147 +This test detects incorrect Git pathspec semantics that mocked argument tests cannot.
149 +- [ ] **Step 3: Verify RED**
151 +Run:
153 +```text
154 +mise run test -- extensions/tune/__tests__/unit.test.ts extensions/tune/__tests__/harness.test.ts
155 +```
157 +Expected: mocked calls observe `bash -c`; the real repository test does not exercise direct Git calls.
159 +- [ ] **Step 4: Implement direct Git discard**
161 +Execute and check these calls sequentially:
163 +```ts
164 +const checkout = await exec("git", [
165 + "checkout",
166 + "--",
167 + ".",
168 + ":(exclude,glob).pi/tune/**",
169 + ":(exclude,glob).pi/tune/*",
170 +], { cwd, timeout: 10000 });
171 +if (checkout.code !== 0) {
172 + throw new Error(
173 + `git checkout failed: ${
174 + (checkout.stdout + checkout.stderr).trim().slice(0, 200)
175 + }`,
176 + );
178 +const clean = await exec("git", [
179 + "clean",
180 + "-fd",
181 + "-e",
182 + ".pi/tune",
183 + "-e",
184 + ".pi/tune/**",
185 +], { cwd, timeout: 10000 });
186 +if (clean.code !== 0) {
187 + throw new Error(
188 + `git clean failed: ${(clean.stdout + clean.stderr).trim().slice(0, 200)}`,
189 + );
191 +```
193 +Keep jj and the existing success text unchanged.
195 +- [ ] **Step 5: Verify GREEN and commit**
197 +```text
198 +mise run test -- extensions/tune/__tests__/unit.test.ts extensions/tune/__tests__/harness.test.ts
199 +git add extensions/tune/index.ts extensions/tune/__tests__/unit.test.ts extensions/tune/__tests__/harness.test.ts
200 +git commit -m "fix(tune): discard through direct git calls"
201 +```
203 +Expected: all focused tests pass.
205 +### Task 3: Characterize and Replace the Benchmark Runner
207 +**Files:**
209 +- Modify: `extensions/tune/__tests__/harness.test.ts`
210 +- Modify: `extensions/tune/__tests__/unit.test.ts`
211 +- Modify: `extensions/tune/__tests__/remaining.test.ts`
212 +- Modify: `extensions/tune/index.ts:145-157,289-299,596-606,1980-2320`
214 +- [ ] **Step 1: Add fixed-schema and real-tool behavior tests**
216 +Assert `run_experiment` has no `command` property.
217 +Through `t.run(when(...calls("run_experiment", params)))`, cover:
219 +- success with stdout and stderr both captured
220 +- primary and secondary `METRIC` parsing
221 +- nonzero benchmark exit
222 +- missing benchmark, called twice with `maxIterations: 1`, never reporting the iteration limit
223 +- checks success
224 +- checks nonzero failure
225 +- checks timeout
226 +- output beyond line and byte limits
227 +- full-output spill file exists and contains early plus late output
229 +Use tiny per-test `benchmark.mjs` and `checks.mjs` fixtures.
230 +For example:
232 +```ts
233 +fs.writeFileSync(
234 + benchmarkPath,
235 + "console.log(`NODE=${process.execPath}`); console.error(\"stderr\"); console.log(\"METRIC score=42\"); console.log(\"METRIC bytes_kb=7\");\n",
237 +await t!.run(when("run", [calls("run_experiment", {}), says("measured")]));
238 +const result = t!.events.toolResultsFor("run_experiment")[0];
239 +expect(result.details).toMatchObject({
240 + exitCode: 0,
241 + passed: true,
242 + parsedPrimary: 42,
243 + parsedMetrics: { score: 42, bytes_kb: 7 },
244 +});
245 +expect(result.text).toContain(`NODE=${process.execPath}`);
246 +expect(result.text).toContain("stderr");
247 +```
249 +- [ ] **Step 2: Add partial-update and abort tests on the registered definition**
251 +Call the same registered definition with `createContext()` only because harness records do not expose `onUpdate` or external cancellation.
252 +A benchmark prints immediately and exits after 1.2 seconds; assert at least one update includes its output.
253 +For abort, use an `AbortController`, abort after spawn, and assert rejection with `aborted`.
255 +- [ ] **Step 3: Add descendant termination tests through the real tool**
257 +Write a benchmark that spawns a child scheduled to create a marker after 300 ms, then remains alive.
258 +Run once with `timeout_seconds: 0.05` through `t.run`, and once through registered-definition execution with an abort signal.
259 +Wait 500 ms after each run and assert the marker never exists.
260 +This proves termination kills descendants, not only the benchmark parent.
262 +- [ ] **Step 4: Verify RED**
264 +Run:
266 +```text
267 +mise run test -- extensions/tune/__tests__/*.test.ts
268 +```
270 +Expected: schema and `{}` tool calls fail against the command-string runner.
272 +- [ ] **Step 5: Remove command strings and launch the fixed MJS file**
274 +Delete `isBenchmarkCommand`, its `__test` export, and its unit/remaining tests.
275 +Remove `command` from `RunParams`.
276 +Require `.pi/tune/benchmark.mjs` before changing runtime state.
277 +Use the fixed display value `node .pi/tune/benchmark.mjs` in runtime and result details.
278 +Launch:
280 +```ts
281 +const child = spawn(process.execPath, [benchmarkPath], {
282 + cwd,
283 + detached: process.platform !== "win32",
284 + windowsHide: true,
285 + stdio: ["ignore", "pipe", "pipe"],
286 +});
287 +```
289 +Keep output accumulation, updates, spill files, timing, parsing, checks, and result construction otherwise unchanged.
290 +Update tool guidance and rendering to show the fixed benchmark and `run_experiment({})`.
292 +- [ ] **Step 6: Implement deterministic single-flight termination**
294 +Change `killTree` to return `Promise<void>` and accept optional platform/taskkill/direct-signal dependencies for controlled tests while production uses real defaults.
295 +On POSIX, retain negative-PID signalling with direct-PID fallback.
296 +On Windows, await `taskkill /F /T /PID <pid>`; call direct signalling only after taskkill spawn error or nonzero close.
298 +Inside the runner, define exactly one termination promise:
300 +```ts
301 +let termination: Promise<void> | null = null;
302 +const terminateOnce = (): Promise<void> => {
303 + if (termination) return termination;
304 + termination = child.pid
305 + ? killTree(child.pid)
306 + : new Promise((resolve) => {
307 + child.once("spawn", () => void killTree(child.pid!).then(resolve));
308 + child.once("error", () => resolve());
309 + });
310 + return termination;
312 +```
314 +Timeout marks `processTimedOut` then calls `void terminateOnce()`.
315 +Abort calls `void terminateOnce()` before or after spawn.
316 +Use one settlement guard for `error` and `close`.
317 +Both paths clear timers/listeners, await termination, close the spill stream, then settle once.
318 +The close path rejects on abort and otherwise resolves the benchmark result.
320 +- [ ] **Step 7: Add Windows taskkill availability and controlled tests**
322 +Resolve `taskkill.exe` once from `SystemRoot\\System32`.
323 +On `session_start`, report one info notification when it is absent; direct child signalling remains fallback.
325 +Unit-test successful taskkill arguments and awaited nonzero fallback with a controlled executable and injected direct-signal callback.
326 +On native Windows, run the real Tune timeout path with a controlled fake taskkill executable on `PATH` or injected resolved path and assert its log is written before the tool returns.
327 +Emit two `session_start` events with an injected missing path and assert only one info notification.
328 +Skip only the native-Windows real-tool fixture on other platforms; helper ordering tests run everywhere.
330 +- [ ] **Step 8: Verify GREEN and commit**
332 +```text
333 +mise run test -- extensions/tune/__tests__/*.test.ts
334 +git add extensions/tune/index.ts extensions/tune/__tests__/harness.test.ts extensions/tune/__tests__/unit.test.ts extensions/tune/__tests__/remaining.test.ts
335 +git commit -m "refactor(tune): run fixed mjs benchmark"
336 +```
338 +Expected: all Tune tests pass, including timeout and abort with no descendant marker.
340 +### Task 4: Use Pi's Node Runtime for Checks and Hooks
342 +**Files:**
344 +- Modify: `extensions/tune/__tests__/unit.test.ts`
345 +- Modify: `extensions/tune/__tests__/harness.test.ts`
346 +- Modify: `extensions/tune/hooks.ts:100-110`
347 +- Modify: `extensions/tune/index.ts:2220-2230`
349 +- [ ] **Step 1: Add PATH-shadow regression tests**
351 +Prepend a controlled fake `node` executable that writes a marker and exits nonzero.
352 +Run a real hook MJS and a passing benchmark plus checks MJS.
353 +Both scripts print `process.execPath`.
354 +Assert hook/check output contains Pi's executable and the fake marker is absent.
355 +Restore `PATH` in `finally`.
356 +This fails against current `spawn("node")` and `pi.exec("node", ...)` calls.
358 +- [ ] **Step 2: Verify RED**
360 +```text
361 +mise run test -- extensions/tune/__tests__/unit.test.ts extensions/tune/__tests__/harness.test.ts
362 +```
364 +Expected: hook or checks reaches the fake PATH executable.
366 +- [ ] **Step 3: Replace PATH-based Node calls**
368 +Use:
370 +```ts
371 +spawn(process.execPath, [script], { cwd: payload.cwd, timeout: TIMEOUT_MS });
372 +```
374 +and:
376 +```ts
377 +pi.exec(process.execPath, [checksPath], {
378 + signal,
379 + timeout: checksTimeout,
380 + cwd,
381 +});
382 +```
384 +- [ ] **Step 4: Verify GREEN and commit**
386 +```text
387 +mise run test -- extensions/tune/__tests__/unit.test.ts extensions/tune/__tests__/harness.test.ts
388 +git add extensions/tune/index.ts extensions/tune/hooks.ts extensions/tune/__tests__/unit.test.ts extensions/tune/__tests__/harness.test.ts
389 +git commit -m "fix(tune): use pi node for mjs scripts"
390 +```
392 +Expected: focused tests pass and the fake PATH node is unused.
394 +### Task 5: Documentation and Repository Verification
396 +**Files:**
398 +- Modify: `extensions/tune/README.md`
399 +- Modify: `extensions/tune/CHANGELOG.md`
400 +- Modify: `CHANGELOG.md`
402 +- [ ] **Step 1: Update documentation**
404 +Document `run_experiment` without a command argument.
405 +State that benchmark, checks, and hooks run with Pi's Node executable.
406 +State that experiments and VCS handling require no Bash, Git Bash, PowerShell, or WSL.
407 +Remove accepted-command and wrapper examples.
408 +Record the breaking tool parameter removal and direct Git discard.
410 +- [ ] **Step 2: Run repository checks**
412 +```text
413 +mise run fmt
414 +mise run lint
415 +mise run test
416 +mise run check
417 +```
419 +Expected: every command exits zero, including the no-Bash static contract added before implementation.
421 +- [ ] **Step 3: Commit documentation and static guard**
423 +```text
424 +git add scripts/test_extensions_static_check.mjs extensions/tune/README.md extensions/tune/CHANGELOG.md CHANGELOG.md
425 +git commit -m "docs(tune): document shell-free runner"
426 +```
428 +### Task 6: Release Gate
430 +**Files:**
432 +- Verify only
434 +- [ ] **Step 1: Run Tune browser E2E**
436 +```text
437 +mise //extensions/tune:e2e
438 +```
440 +Expected: Chromium and Firefox dashboard tests pass.
442 +- [ ] **Step 2: Run full CI**
444 +```text
445 +mise run ci
446 +```
448 +Expected: check, lint, test, and E2E exit zero.
450 +- [ ] **Step 3: Inspect final state**
452 +```text
453 +git status --short --branch
454 +git log --oneline -10
455 +```
457 +Expected: clean implementation worktree and focused commits for the spec, plan, Git discard, MJS runner, runtime consistency, and documentation.

docs/super/specs/2026-07-16-tune-cross-platform-runner-design.md

Modified
125 125 On macOS and Linux, create a detached process group and signal the group by negative PID, retaining the current behavior.
126 126 On Windows, launch without a detached console and invoke the OS-provided `taskkill /F /T /PID <pid>` command.
128 -Windows termination treats `taskkill` as the primary operation and awaits its exit before completing timeout or abort handling.
128 +Windows termination resolves `taskkill.exe` from `SystemRoot\\System32` when the extension loads.
129 +If it is unavailable, Tune emits one informational notification and retains direct child signalling as fallback.
130 +When available, Tune treats `taskkill` as the primary operation and awaits its exit before completing timeout or abort handling.
129 131 Only a spawn error or nonzero `taskkill` exit falls back to signalling the benchmark process directly.
130 132 This ordering prevents killing the parent before `taskkill` discovers its descendants.
223 225 - Tune production code never invokes Bash.
224 226 - Tune does not require WSL, Git Bash, PowerShell, or another command shell for experiments or VCS handling.
227 +- Windows reports missing `taskkill.exe` once at extension load and retains direct child termination fallback.
225 228 - `run_experiment` always executes `.pi/tune/benchmark.mjs` with `process.execPath`.
226 229 - Windows, macOS, and Linux expose the same `run_experiment` tool schema and benchmark behavior.
227 230 - Benchmark success, failure, timeout, abort, output, metrics, and checks retain current behavior.