1 +#
Overkill Skill Training Design 5 +Add `overkill`, a native TypeScript Pi extension that improves an existing Pi Markdown skill without changing model weights.
7 +It ports SkillOpt's useful concepts into Pi rather than embedding SkillOpt or reproducing its Python framework.
9 +`overkill` is a normal stateful Pi command controller, not an Ultra workflow.
11 +The name is deliberately excessive: skill level over 9000.
15 +| SkillOpt concept | Overkill feature | Existing extension example |
17 +| Trainable skill document | Existing Pi skill selected by path | Pi resource loader |
18 +| Target rollout | Fresh isolated Pi sub-agent per task | Ultra |
19 +| Optimizer | Pi model completion that returns structured Markdown patches | Ultra |
20 +| Task evaluator | Project-supplied checker adapter returning a numeric score | Tune |
21 +| Held-out gate | Strict validation-score accept/reject | Tune |
22 +| Epoch/run state | Append-only custom Pi session entries, rebuilt on resume | Goal |
23 +| Long artifacts | Private platform state files addressed from session entries | llmiterate |
24 +| Final deploy | Pi interactive select/confirm before any skill mutation | Pi extension UI |
28 +- No Python dependency, migration, or external runtime.
29 +- No Ultra workflow, dashboard, watcher, background daemon, database, or new package dependency.
30 +- No model-weight training.
31 +- No automatic overwrite of a skill.
32 +- No non-Pi execution backends in version one.
33 +- No reuse of Tune's VCS rollback loop.
38 +/overkill <skill-path> <suite-dir>
44 +`/overkill <skill-path> <suite-dir>` validates the existing Pi skill and a task suite, then starts a run.
46 +`/overkill status` reports current stage, best validation score, completed work, and a compact failure summary.
48 +`/overkill resume` continues the current session branch's interrupted run.
50 +`/overkill discard` deletes its private artifacts and appends a terminal session entry.
52 +Overkill has no model-callable tools in version one.
54 +The user starts and controls training; Pi sub-agents perform task rollouts.
56 +##
Task Suite And Evaluator 58 +The caller supplies a suite directory containing explicit splits:
68 +Each JSON item has an `id` and a task `prompt`.
70 +`check.mjs` receives one JSON object on standard input containing the task item and target-agent result.
72 +It writes one JSON object with one finite numeric `score` to standard output.
74 +Higher score is better in version one.
76 +This preserves SkillOpt's evaluator abstraction while using Tune's familiar checker pattern.
78 +Malformed tasks, checker timeouts, nonzero exits, invalid JSON, or invalid scores fail that item and become recorded evidence; they never accept a candidate.
80 +The suite remains user-owned input and is never generated or modified by Overkill.
82 +##
Sub-Agent Rollouts 84 +Overkill uses Ultra's isolated-sub-agent pattern directly, not Ultra's workflow DSL:
86 +- fresh in-memory session manager per task;
87 +- fresh resource loader per task;
88 +- explicit target-model and tool allowlists;
89 +- host skills, context files, prompt templates, themes, and extensions disabled by default;
90 +- candidate skill appended through the isolated loader's system-prompt override;
91 +- disposal and cancellation cleanup after every task.
93 +The target model stays fixed for a run.
95 +Train and validation rollouts may fan out up to the configured concurrency limit, as Ultra does.
97 +The first version defaults to one worker for reproducibility.
99 +Overkill records bounded assistant output and tool-result tails for the optimizer and checker, never whole session histories.
101 +##
Optimizer And Gate 103 +After a train batch, the selected optimizer model receives bounded rollout evidence plus the current candidate skill.
105 +It returns structured add, replace, and delete patches.
107 +Overkill validates patch targets, rejects malformed or overlapping edits, and applies at most the configured edit budget.
109 +The candidate then runs against `val/`.
111 +Overkill accepts it only when its validation score is strictly greater than the current candidate's score.
113 +It separately retains the best accepted candidate.
115 +The `test/` split is used once, after training, to report the best candidate's final score.
117 +It never controls patch acceptance.
119 +##
Persistence Without Project Litter 121 +Overkill requires persistence only for status, pause, and resume.
123 +It stores small, branch-aware state as append-only custom Pi session entries: run identity, source-skill fingerprint, suite fingerprint, stage, scores, candidate version, artifact references, and terminal decision.
125 +It reconstructs state by replaying the current session branch, following Goal's pattern.
127 +Candidate files, patch proposals, bounded rollout evidence, and checkpoints live outside the project in a private platform state directory:
129 +- `$XDG_STATE_HOME/pi/overkill/` when explicitly set;
130 +- `%LOCALAPPDATA%/pi/overkill/` on Windows;
131 +- `~/Library/Application Support/pi/overkill/` on macOS;
132 +- `~/.local/state/pi/overkill/` on Linux.
134 +Run directories use a project-root hash and run ID, matching llmiterate's cross-platform state-store approach.
136 +The user can resume only from the originating session branch.
138 +Overkill deletes private artifacts after explicit discard or completed adoption, unless a future retention setting is added.
140 +The source skill and project directory stay untouched until final adoption.
144 +After the final test, interactive Pi presents three choices:
146 +1. overwrite the original skill;
147 +2. write a new user-chosen skill path;
150 +The selected write uses Pi's file-mutation queue.
152 +RPC, JSON, print, and other non-interactive contexts stage the best candidate in platform state and report its path; they never write a final skill.
154 +##
Extension Shape And Style 156 +`extensions/overkill/` follows the existing extension style: ESM TypeScript, `index.ts`, focused pure modules, package metadata, README, Vitest tests, and root `mise` tasks.
158 +It uses Node built-ins, Pi APIs, and installed repository dependencies only.
160 +During a run, Overkill uses a Goal-like footer status and above-editor widget: stage, completed / failed / queued counts, current and best validation scores, and one bounded failure line.
162 +In TUI the widget uses a theme-aware component factory; in RPC it falls back to string lines.
164 +Final adoption uses Pi's native `select`, `input`, and `confirm` dialogs, which also work through RPC's UI protocol.
166 +Overkill does not use an Ultra overlay, Tune dashboard, custom editor, or `ctx.ui.custom()` component: training needs aggregate progress, not per-agent inspection or steering.
170 +A source skill must resolve to an existing Markdown file within the trusted project context or configured Pi skill directories.
172 +A suite must be readable and valid before any model call.
174 +Missing model authentication, invalid paths, checker failure, malformed optimizer output, cancellation, or a failed sub-agent stop the affected work without modifying the source skill.
176 +Cancellation preserves private artifacts for `/overkill resume`.
178 +Only explicit discard or final adoption deletes them.
182 +- Unit tests: suite parsing, checker-result validation, strict gate logic, patch validation/application, platform state paths, session-entry replay, and adoption staging.
183 +- SDK-driver tests: isolated Pi-session construction, resource isolation, tool allowlists, cancellation, and cleanup, following Ultra.
184 +- Harness test: load in a real Pi extension session.
185 +- Deterministic end-to-end fixture: fixed target and optimizer adapters prove train → patch → validation gate → final-test → staged adoption without live model calls.
186 +- UI tests: compact state and expanded bounded diagnostics.
188 +##
Acceptance Criteria 190 +- Overkill evaluates a named existing Pi skill against train, validation, and test splits.
191 +- Rollouts use isolated Pi sub-agents, not Ultra workflows or external executors.
192 +- A candidate is accepted only on strict validation improvement.
193 +- Test data never selects candidates.
194 +- No project-local Overkill state files are created.
195 +- Session entries support branch-correct resume; private artifacts work on Windows, macOS, and Linux.
196 +- The source skill changes only after an explicit interactive adoption choice.
197 +- Invalid inputs and failures leave the source skill unchanged.
198 +- `mise run ci` remains the release gate.