repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

docs(tune): design cross-platform runner

9cd93c168db36d5e73d2e36e2e3cc08c31e41d7e

Oliver Krylow <o.krylow@isp-insoft.de> ·

unsigned

Browse filesCompare with HEAD

Changed files (2)

CHANGELOG.md

Modified
2 2 # Changelog
4 4 ## Unreleased
6 +- Add a Tune cross-platform runner design that removes Bash/WSL and shell command strings, runs `.pi/tune/benchmark.mjs` directly with Node on every platform, preserves existing experiment behavior, and uses direct Git calls for discard
6 7 - Remove the `mode`, `shame`, `ssh-tools`, and `terminal-signals` extensions
7 8 - Add an `/overkill aftermath` design for improving one confirmed skill from all past Pi sessions, with checkable task mining, deterministic checks plus LLM rubric fallback, hash-split validation, strict no-tie acceptance, resumable staged results, and no automatic adoption or scheduling
8 9 - Fix npm audit's moderate `protobufjs` vulnerability by updating the root lockfile entry to 7.6.5

docs/super/specs/2026-07-16-tune-cross-platform-runner-design.md

Added
new file mode 100644
1 +# Tune Cross-Platform Runner
3 +## Goal
5 +Remove Tune's Bash and WSL dependency without changing its experiment lifecycle, persisted state, metric handling, checks, VCS decisions, dashboards, or autonomous loop.
7 +Tune uses one benchmark contract on Windows, macOS, and Linux: execute `.pi/tune/benchmark.mjs` with the Node executable running Pi.
9 +## Current Behavior
11 +Tune already standardizes user-authored automation on JavaScript modules:
13 +- `.pi/tune/benchmark.mjs` measures the experiment.
14 +- `.pi/tune/checks.mjs` validates a passing benchmark.
15 +- `.pi/tune/hooks/before.mjs` and `.pi/tune/hooks/after.mjs` steer iterations.
17 +Bash remains in two production paths:
19 +- `run_experiment` executes a command string through `bash -c`.
20 +- Git discard executes `git checkout` and `git clean` through a Bash script.
22 +The command guard accepts a narrow Bash-shaped benchmark command when `benchmark.mjs` exists, including environment prefixes and `env`, `time`, `nice`, or `nohup` wrappers.
23 +Without `benchmark.mjs`, it permits arbitrary shell commands.
25 +This shell surface is the WSL dependency on Windows.
26 +It is not required by Tune's experiment model.
28 +## Scope
30 +Change only experiment process launch, process termination, Git discard, related tool text, tests, and documentation.
32 +Preserve:
34 +- `/tune` commands and autonomous turn scheduling
35 +- `init_experiment` and `log_experiment` parameters
36 +- benchmark timing and default timeout
37 +- merged stdout and stderr capture
38 +- live partial output
39 +- bounded LLM output and full-output spill files
40 +- `METRIC name=value` parsing
41 +- checks behavior and timeout
42 +- keep, discard, crash, and checks-failed statuses
43 +- Git and jj decisions
44 +- `.pi/tune` lock and all persisted files
45 +- confidence calculations
46 +- TUI and browser dashboards
47 +- compaction and hooks
49 +Remove:
51 +- the `run_experiment.command` parameter
52 +- arbitrary experiment command strings
53 +- environment-prefix parsing
54 +- `env`, `time`, `nice`, and `nohup` wrappers
55 +- every Tune production invocation of `bash`
57 +Users needing custom environment setup, subprocesses, or timing wrappers put that logic in `benchmark.mjs` using Node APIs.
59 +## Non-Goals
61 +- Supporting arbitrary shell commands through platform-native shells
62 +- Emulating Bash syntax in TypeScript
63 +- Preserving command-string compatibility
64 +- Adding a process-runner dependency
65 +- Refactoring unrelated parts of the large `index.ts`
66 +- Changing benchmark or session file formats
67 +- Changing dashboard browser launching
69 +## Rejected Approaches
71 +### Resolve Bash differently on Windows
73 +Git Bash could replace WSL while preserving shell command strings.
74 +This retains an unnecessary shell dependency and leaves Tune with two scripting contracts.
76 +### Select Bash, PowerShell, or `cmd.exe` by platform
78 +Native shells have different quoting, environment, chaining, and error semantics.
79 +This would make the same Tune input behave differently across platforms.
81 +### Implement a portable shell parser
83 +A custom parser would be larger and less reliable than the experiment runner it replaces.
84 +The supported benchmark is already an MJS file, so a shell language adds no essential capability.
86 +## Design
88 +### Experiment tool contract
90 +`run_experiment` accepts only optional timeout fields:
92 +- `timeout_seconds`
93 +- `checks_timeout_seconds`
95 +The tool requires `.pi/tune/benchmark.mjs`.
96 +If the file is absent, it returns a clear tool error instructing the agent to create it.
98 +The system prompt and tool guidance tell the agent to edit `benchmark.mjs`, then call `run_experiment` without a command.
100 +Call and result rendering display `.pi/tune/benchmark.mjs` as the fixed experiment target.
101 +The result details may retain a fixed display command for internal UI continuity, but callers cannot choose it.
103 +### Node process launch
105 +Launch the benchmark directly with:
107 +- executable: `process.execPath`
108 +- argument: the absolute `.pi/tune/benchmark.mjs` path
109 +- working directory: the Pi project directory
110 +- shell: disabled
111 +- stdin: ignored
112 +- stdout and stderr: piped
113 +- Windows console: hidden
115 +Using `process.execPath` guarantees the benchmark uses the same Node installation as Pi and avoids `PATH` differences.
117 +Keep the existing output accumulator, update interval, truncation, spill-file, metric parsing, timeout accounting, and result construction unchanged around the new child process.
119 +Run checks and hook modules with `process.execPath` for the same runtime guarantee.
121 +### Timeout and abort
123 +Preserve process-tree termination rather than killing only the benchmark parent.
125 +On macOS and Linux, create a detached process group and signal the group by negative PID, retaining the current behavior.
126 +On Windows, launch without a detached console and invoke the OS-provided `taskkill /F /T /PID <pid>` command.
128 +Windows termination treats `taskkill` as the primary operation and awaits its exit before completing timeout or abort handling.
129 +Only a spawn error or nonzero `taskkill` exit falls back to signalling the benchmark process directly.
130 +This ordering prevents killing the parent before `taskkill` discovers its descendants.
132 +This is the only platform-specific experiment-runner branch.
133 +It implements one external timeout and abort contract on all platforms.
134 +It does not affect benchmark command semantics because no shell interprets benchmark input.
136 +Tune reports timeout or abort through the existing result path.
138 +### Git discard
140 +Replace the Bash script with two sequential `pi.exec` calls using argument arrays:
142 +1. `git checkout -- . :(exclude,glob).pi/tune/** :(exclude,glob).pi/tune/*`
143 +2. `git clean -fd -e .pi/tune -e .pi/tune/**`
145 +Git, not a shell, interprets the pathspec and exclusions.
146 +The arguments therefore have the same meaning on every platform.
148 +Check each exit code.
149 +If checkout fails, do not run clean.
150 +If clean fails, report the bounded Git error.
151 +Never report a successful discard unless both commands succeed.
153 +The jj discard path remains unchanged.
155 +### Structure
157 +Keep the implementation in `extensions/tune/index.ts` to minimize movement in known-working code.
158 +Do not introduce a runner interface, backend factory, compatibility adapter, or new production module.
160 +Reuse existing helpers and test exports where practical.
161 +Delete `isBenchmarkCommand` and its tests because no command string remains.
163 +Update `extensions/tune/README.md` and `extensions/tune/CHANGELOG.md` with the new fixed benchmark contract and removed Bash dependency.
165 +## Error Handling
167 +A missing benchmark file fails before process creation and does not increment experiment counts.
169 +A benchmark spawn failure remains a failed tool execution with the executable and bounded error visible.
170 +A nonzero benchmark exit remains a crash result.
171 +A timeout remains distinguishable from a normal nonzero exit.
172 +An abort stops the process tree and follows Pi's existing aborted-tool behavior.
174 +Checks run only after a passing benchmark, unchanged.
175 +A checks failure still blocks `keep`.
177 +A Git discard failure remains visible and does not claim the checkout was restored.
178 +Tune preserves `.pi/tune` throughout discard attempts.
180 +No failure changes or truncates existing session history.
182 +## Regression Strategy
184 +Add characterization coverage before changing production behavior.
185 +The characterization assertions become the contract for the direct Node runner.
187 +Cover through the real Tune tool path where the harness permits:
189 +- benchmark success and nonzero exit
190 +- timeout and abort
191 +- descendant-process termination
192 +- interleaved stdout and stderr capture
193 +- live partial updates
194 +- line and byte truncation
195 +- full-output spill files
196 +- primary and secondary metric parsing
197 +- checks success, failure, and timeout
198 +- no experiment-count increment when the benchmark is missing
199 +- Git keep
200 +- Git discard with `.pi/tune` preserved
201 +- Git checkout failure preventing `git clean`
202 +- jj keep and discard remaining unchanged
203 +- hooks using the same Node executable
205 +Use temporary repositories and temporary `.pi/tune` fixtures.
206 +Use controlled fake platform binaries on `PATH` for Git, jj, and Windows termination command behavior.
207 +Do not limit coverage to mocked helper calls.
209 +Keep existing JSONL, compaction, dashboard, settings, shortcut, and browser tests unchanged.
210 +Their continued success guards unrelated behavior.
212 +Run the repository tasks through Mise:
214 +- `mise run fmt`
215 +- `mise run lint`
216 +- `mise run test`
217 +- `mise run check`
218 +- `mise run e2e`
219 +- `mise run ci`
221 +## Acceptance Criteria
223 +- Tune production code never invokes Bash.
224 +- Tune does not require WSL, Git Bash, PowerShell, or another command shell for experiments or VCS handling.
225 +- `run_experiment` always executes `.pi/tune/benchmark.mjs` with `process.execPath`.
226 +- Windows, macOS, and Linux expose the same `run_experiment` tool schema and benchmark behavior.
227 +- Benchmark success, failure, timeout, abort, output, metrics, and checks retain current behavior.
228 +- Git and jj keep/discard behavior retains current outcomes.
229 +- `.pi/tune` survives every discard path.
230 +- Existing session files continue without migration.
231 +- No new runtime dependency or abstraction is added.
232 +- `mise run ci` passes.
234 +## Known Boundary
236 +Node cannot terminate descendant process trees identically through one standard-library call on every operating system.
237 +The design isolates that unavoidable platform difference to process termination while keeping the public experiment contract identical.