repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

fix(goal): show judge history and retry empty verdicts

2c556d42ee8f9d970d92b98be1d18d6d28f083c6

Oliver Jan Krylow <oliver@bugabinga.net> ·

unsigned

Browse filesCompare with HEAD

Changed files (7)

extensions/goal/CHANGELOG.md

Modified
12 12 - Persist `goal-state` entries; replay newest state or clear tombstone.
13 13 - Run judge after each turn; pause on budget, complete/block on verdict.
14 14 - Gate `update_goal` until judge review; enforce session ownership.
15 -- Show widget/status/footer progress; autocomplete subcommands and `/goal remove`.
15 +- Show widget/status/footer progress; autocomplete subcommands and `/goal
16 + remove`.

extensions/goal/README.md

Modified
1 1 # goal
3 3 <!-- demo:goal:start -->
4 5 ![Goal loop](assets/goal.gif)
5 7 <!-- demo:goal:end -->
7 9 Persistent standing-goal loop for Pi sessions.
9 11 ## Install / load
11 -Loaded through the root pi-ext package. See [../../README.md](../../README.md).
13 +Loaded through the root pi-ext package.
14 +See [../../README.md](../../README.md).
13 16 ## Commands
15 -Subcommands autocomplete. `/goal remove` autocompletes criterion numbers.
18 +Subcommands autocomplete.
19 +`/goal remove` autocompletes criterion numbers.
17 21 ```text
18 22 /goal
40 44 update_goal({ status: "complete" | "blocked", reason: string })
43 -Marks the active goal complete or blocked. Hidden until a continue verdict increments `turnsUsed`, then hidden again when the goal is complete, blocked, paused, cleared, or resumed into a fresh 0-turn state. If invoked while stale, it errors when no active goal exists, the goal has not been judged, or the goal belongs to another session.
47 +Marks the active goal complete or blocked.
48 +Hidden until a continue verdict increments `turnsUsed`, then hidden again when
49 +the goal is complete, blocked, paused, cleared, or resumed into a fresh 0-turn
50 +state.
51 +If invoked while stale, it errors when no active goal exists, the goal has not
52 +been judged, or the goal belongs to another session.
45 54 ## Settings
57 -- `maxTurns`: turn budget for a new goal. Default `20`, clamped to `1..10000`.
58 -- `judgeModel`: optional Pi model ref, `provider/model`. When unset, the current session model is used.
59 -- `tailBytes`: bytes read from the latest assistant response for judgment. Default `4096`, clamped to `512..65536`.
66 +- `maxTurns`:
67 + turn budget for a new goal.
68 + Default `20`, clamped to `1..10000`.
69 +- `judgeModel`:
70 + optional Pi model ref, `provider/model`.
71 + When unset, the current session model is used.
72 +- `tailBytes`:
73 + bytes read from the latest assistant response for judgment.
74 + Default `4096`, clamped to `512..65536`.
61 76 ## Hooks
71 86 ## Behavior
73 88 - `/goal` and `/goal status` print current goal status.
74 -- `/goal <objective>` creates an active goal with empty criteria, current session ownership, and the configured turn budget. It sends a kickoff prompt immediately.
75 -- A raw prompt containing `standing goal` plus an `<objective>...</objective>` block is adopted as an active goal when no goal exists.
76 -- New goals require an idle agent. Subcommands can run while the agent is busy.
77 -- Goal state is stored as append-only custom `goal-state` entries. Replay scans backward; newest valid state or clear tombstone wins.
78 -- `before_agent_start` injects the active goal and criteria into the system prompt for the owning session.
79 -- After each owned active goal turn, the extension writes a `goal-judge` start entry, shows `goal judging N/M` in persistent UI, then asks a judge model for strict JSON: `done`, `blocked`, `reason`.
80 -- The judge sees the objective, criteria, and tail of the latest assistant response.
81 -- Judge completion writes a `goal-judge` verdict entry and updates the goal widget/footer with the latest verdict reason.
82 -- Continue verdicts stay quiet except for persistent UI. Complete, blocked, paused, and judge-failed verdicts notify.
83 -- Done verdict marks the goal complete and clears persistent status/widget/footer. Blocked verdict marks it blocked and keeps persistent UI.
89 +- `/goal <objective>` creates an active goal with empty criteria, current
90 + session ownership, and the configured turn budget.
91 + It sends a kickoff prompt immediately.
92 +- A raw prompt containing `standing goal` plus an `<objective>...</objective>`
93 + block is adopted as an active goal when no goal exists.
94 +- New goals require an idle agent.
95 + Subcommands can run while the agent is busy.
96 +- Goal state is stored as append-only custom `goal-state` entries.
97 + Replay scans backward; newest valid state or clear tombstone wins.
98 +- `before_agent_start` injects the active goal and criteria into the system
99 + prompt for the owning session.
100 +- After each owned active goal turn, the extension writes a `goal-judge` start
101 + entry, shows `goal judging N/M` in persistent UI, then asks a judge model for
102 + strict JSON:
103 + `done`, `blocked`, `reason`.
104 +- The judge sees the objective, criteria, and tail of the latest assistant
105 + response.
106 +- Judge completion writes a `goal-judge` verdict entry and updates the goal
107 + widget/footer with the latest verdict reason.
108 +- Continue verdicts stay quiet except for persistent UI.
109 + Complete, blocked, paused, and judge-failed verdicts notify.
110 +- Done verdict marks the goal complete and clears persistent
111 + status/widget/footer.
112 + Blocked verdict marks it blocked and keeps persistent UI.
84 113 - Incomplete verdict increments `turnsUsed`; exhausted budget pauses the goal.
85 -- If still active, continuation is sent when the session becomes idle. Retry window: 20 attempts, 250 ms apart after the immediate attempt.
86 -- `update_goal` is hidden until the first continue verdict increments `turnsUsed`, then hidden again for terminal or paused goals.
114 +- If still active, continuation is sent when the session becomes idle.
115 + Retry window:
116 + 20 attempts, 250 ms apart after the immediate attempt.
117 +- `update_goal` is hidden until the first continue verdict increments
118 + `turnsUsed`, then hidden again for terminal or paused goals.
87 119 - `/goal pause` only pauses active owned goals.
88 -- `/goal resume` rebinds a goal to the current session, sets it active, resets `turnsUsed`, and schedules continuation.
89 -- `/goal complete [reason]` is a user override. Default reason: `marked complete by user`.
120 +- `/goal resume` rebinds a goal to the current session, sets it active, resets
121 + `turnsUsed`, and schedules continuation.
122 +- `/goal complete [reason]` is a user override.
123 + Default reason:
124 + `marked complete by user`.
90 125 - `/goal clear` writes a clear tombstone and removes status.
91 -- `/goal add`, `/goal remove`, and `/goal clear-criteria` edit criteria on the current owned goal.
92 -- `/goal criteria` lists criteria. Empty criteria print `No goal criteria.`
93 -- UI status key: `goal` for active, paused, judging, and blocked goals when the shared `footer` extension is absent. Complete/cleared goals clear status and notify instead.
94 -- UI widget key: `goal`. Active, judging, paused, and blocked goals show an above-editor widget with phase, turn progress, objective, and last judge reason.
95 -- Shared footer integration: emits `footer:segment` id `goal` when `extensions/footer` is loaded.
96 -- Goals owned by another session show `owned by another session`; mutating commands require `/goal resume` first.
97 -- `PI_GOAL_PROBE=1` adds `/goal-probe` and writes custom `goal-probe` entries for continuation scheduling checks.
126 +- `/goal add`, `/goal remove`, and `/goal clear-criteria` edit criteria on the
127 + current owned goal.
128 +- `/goal criteria` lists criteria.
129 + Empty criteria print `No goal criteria.`
130 +- UI status key:
131 + `goal` for active, paused, judging, and blocked goals when the shared `footer`
132 + extension is absent.
133 + Complete/cleared goals clear status and notify instead.
134 +- UI widget key:
135 + `goal`.
136 + Active, judging, paused, and blocked goals show an above-editor widget with
137 + phase, turn progress, objective, and last judge reason.
138 +- Shared footer integration:
139 + emits `footer:segment` id `goal` when `extensions/footer` is loaded.
140 +- Goals owned by another session show `owned by another session`; mutating
141 + commands require `/goal resume` first.
142 +- `PI_GOAL_PROBE=1` adds `/goal-probe` and writes custom `goal-probe` entries
143 + for continuation scheduling checks.

extensions/goal/__tests__/harness.test.ts

Modified
1 1 import { afterEach, describe, expect, it } from "bun:test";
2 -import { createAssistantMessageEventStream, getApiProvider, registerApiProvider } from "@earendil-works/pi-ai";
3 -import { createTestSession, type TestSession } from "@marcfargas/pi-test-harness";
2 +import {
3 + createAssistantMessageEventStream,
4 + getApiProvider,
5 + registerApiProvider,
6 +} from "@earendil-works/pi-ai";
7 +import {
8 + createTestSession,
9 + type TestSession,
10 +} from "@marcfargas/pi-test-harness";
4 11 import { __goalTest, GOAL_PROBE_MESSAGE } from "../index.ts";
6 13 let t: TestSession | undefined;
52 -async function waitFor(predicate: () => boolean, timeoutMs = 500): Promise<void> {
59 +async function waitFor(
60 + predicate: () => boolean,
61 + timeoutMs = 500,
62 +): Promise<void> {
53 63 const start = Date.now();
54 64 while (!predicate()) {
55 - if (Date.now() - start > timeoutMs) throw new Error("timed out waiting for condition");
65 + if (Date.now() - start > timeoutMs)
66 + throw new Error("timed out waiting for condition");
56 67 await new Promise((resolve) => setTimeout(resolve, 5));
60 71 function customEvents(kind: string): Array<Record<string, unknown>> {
61 - return (t!.session.sessionManager.getEntries() as Array<{ type: string; customType?: string; data?: Record<string, unknown> }>)
72 + return (
73 + t!.session.sessionManager.getEntries() as Array<{
74 + type: string;
75 + customType?: string;
76 + data?: Record<string, unknown>;
77 + }>
78 + )
62 79 .filter((entry) => entry.type === "custom" && entry.customType === kind)
63 80 .map((entry) => entry.data ?? {});
83 +function customMessages(kind: string): Array<Record<string, unknown>> {
84 + return (
85 + t!.session.sessionManager.getEntries() as Array<{
86 + type: string;
87 + customType?: string;
88 + content?: string;
89 + display?: boolean;
90 + details?: Record<string, unknown>;
91 + }>
92 + )
93 + .filter(
94 + (entry) => entry.type === "custom_message" && entry.customType === kind,
95 + )
96 + .map((entry) => ({
97 + content: entry.content,
98 + display: entry.display,
99 + details: entry.details,
100 + }));
66 103 function latestGoal() {
67 - return __goalTest.latestGoalState({ sessionManager: t!.session.sessionManager } as never);
104 + return __goalTest.latestGoalState({
105 + sessionManager: t!.session.sessionManager,
106 + } as never);
70 109 function installJudgeProvider(replies: string[]) {
71 110 const previousProvider = getApiProvider("openai-responses");
72 111 const queue = [...replies];
73 - registerApiProvider({
74 - api: "openai-responses",
75 - stream: () => {
76 - const stream = createAssistantMessageEventStream();
77 - queueMicrotask(() => stream.push(doneEvent(queue.shift() ?? '{"done":false,"blocked":false,"reason":"exhausted"}')));
78 - return stream;
112 + registerApiProvider(
114 + api: "openai-responses",
115 + stream: () => {
116 + const stream = createAssistantMessageEventStream();
117 + queueMicrotask(() =>
118 + stream.push(
119 + doneEvent(
120 + queue.shift() ??
121 + '{"done":false,"blocked":false,"reason":"exhausted"}',
122 + ),
123 + ),
124 + );
125 + return stream;
126 + },
127 + streamSimple: () => {
128 + throw new Error("not used");
129 + },
80 - streamSimple: () => {
81 - throw new Error("not used");
82 - },
83 - }, "goal-test");
131 + "goal-test",
132 + );
84 133 return () => {
85 - if (previousProvider) registerApiProvider(previousProvider, "goal-test-restore");
134 + if (previousProvider)
135 + registerApiProvider(previousProvider, "goal-test-restore");
89 -async function runProbe(mode: "direct" | "microtask" | "timeout", expectedAgentEnds: number) {
139 +async function runProbe(
140 + mode: "direct" | "microtask" | "timeout",
141 + expectedAgentEnds: number,
142 +) {
90 143 process.env.PI_GOAL_PROBE = "1";
91 144 t = await createTestSession({ extensions: [goalPath()] });
92 145 installSaysStream(t, ["first turn", "second turn"]);
94 147 await t.session.prompt(`/goal-probe ${mode}`);
95 148 await t.session.prompt("initial work");
96 - await waitFor(() => t!.events.all.filter((event) => event.type === "agent_end").length >= expectedAgentEnds);
149 + await waitFor(
150 + () =>
151 + t!.events.all.filter((event) => event.type === "agent_end").length >=
152 + expectedAgentEnds,
153 + );
97 154 await new Promise((resolve) => setTimeout(resolve, 25));
98 155 await t.session.agent.waitForIdle();
101 158 describe("goal integration", () => {
102 159 it("update_goal is unavailable before judge review", async () => {
103 - t = await createTestSession({ extensions: [goalPath()], propagateErrors: false });
160 + t = await createTestSession({
161 + extensions: [goalPath()],
162 + propagateErrors: false,
163 + });
104 164 await t.run({
105 165 prompt: "/goal ship it",
106 166 actions: [
107 - { type: "call", toolName: "update_goal", params: { status: "complete", reason: "verified" } },
168 + type: "call",
169 + toolName: "update_goal",
170 + params: { status: "complete", reason: "verified" },
171 + },
108 172 { type: "say", text: "done" },
112 176 const result = t.events.toolResultsFor("update_goal")[0];
113 177 expect(result.isError).toBe(true);
114 178 expect(result.text).toContain("Tool update_goal not found");
115 - expect(latestGoal()).toMatchObject({ objective: "ship it", status: "active" });
179 + expect(latestGoal()).toMatchObject({
180 + objective: "ship it",
181 + status: "active",
182 + });
118 185 it("update_goal is unavailable without active judged goal", async () => {
119 - t = await createTestSession({ extensions: [goalPath()], propagateErrors: false });
186 + t = await createTestSession({
187 + extensions: [goalPath()],
188 + propagateErrors: false,
189 + });
120 190 await t.run({
121 191 prompt: "try tool",
122 - actions: [{ type: "call", toolName: "update_goal", params: { status: "complete", reason: "nothing active" } }],
192 + actions: [
194 + type: "call",
195 + toolName: "update_goal",
196 + params: { status: "complete", reason: "nothing active" },
197 + },
198 + ],
125 201 const result = t.events.toolResultsFor("update_goal")[0];
130 206 it("/goal complete is a user override and updates passive status", async () => {
131 - const restore = installJudgeProvider(['{"done":false,"blocked":false,"reason":"not yet"}']);
207 + const restore = installJudgeProvider([
208 + '{"done":false,"blocked":false,"reason":"not yet"}',
209 + ]);
133 211 t = await createTestSession({ extensions: [goalPath()] });
134 212 installSaysStream(t, ["partial"]);
137 215 await waitFor(() => latestGoal()?.status === "active");
138 216 await t.session.prompt("/goal complete accepted by user");
140 - expect(latestGoal()).toMatchObject({ objective: "manually finish", status: "complete", lastReason: "accepted by user" });
218 + expect(latestGoal()).toMatchObject({
219 + objective: "manually finish",
220 + status: "complete",
221 + lastReason: "accepted by user",
222 + });
141 223 const statusTexts = t.events.ui
142 - .filter((call) => call.method === "setStatus" && call.args[0] === "goal")
224 + .filter(
225 + (call) => call.method === "setStatus" && call.args[0] === "goal",
143 227 .map((call) => call.args[1]);
144 228 expect(statusTexts).toContain("goal active 0/20");
145 229 expect(statusTexts).toContain(undefined);
151 235 it("adopts raw standing-goal prompts and runs judge", async () => {
152 - const restore = installJudgeProvider(['{"done":true,"blocked":false,"reason":"raw standing goal complete"}']);
236 + const restore = installJudgeProvider([
237 + '{"done":true,"blocked":false,"reason":"raw standing goal complete"}',
238 + ]);
154 240 t = await createTestSession({ extensions: [goalPath()] });
155 241 installSaysStream(t, ["shipped the raw goal"]);
157 - await t.session.prompt("Start working toward this standing goal.\n\n<objective>\nship raw prompt\n</objective>\n\nDo not stop just because one turn ends.");
243 + await t.session.prompt(
244 + "Start working toward this standing goal.\n\n<objective>\nship raw prompt\n</objective>\n\nDo not stop just because one turn ends.",
245 + );
158 246 await waitFor(() => latestGoal()?.status === "complete");
160 - expect(latestGoal()).toMatchObject({ objective: "ship raw prompt", status: "complete", lastReason: "raw standing goal complete" });
248 + expect(latestGoal()).toMatchObject({
249 + objective: "ship raw prompt",
250 + status: "complete",
251 + lastReason: "raw standing goal complete",
252 + });
161 253 expect(customEvents("goal-judge")).toMatchObject([
162 254 { event: "start", objective: "ship raw prompt", turn: 1, maxTurns: 20 },
163 - { event: "verdict", objective: "ship raw prompt", turn: 1, maxTurns: 20, verdict: { done: true, blocked: false, reason: "raw standing goal complete" } },
256 + event: "verdict",
257 + objective: "ship raw prompt",
258 + turn: 1,
259 + maxTurns: 20,
260 + verdict: {
261 + done: true,
262 + blocked: false,
263 + reason: "raw standing goal complete",
264 + },
265 + },
165 267 } finally {
166 268 restore();
170 272 it("judge complete verdict completes goal", async () => {
171 - const restore = installJudgeProvider(['{"done":true,"blocked":false,"reason":"all criteria verified"}']);
273 + const restore = installJudgeProvider([
274 + '{"done":true,"blocked":false,"reason":"all criteria verified"}',
275 + ]);
173 277 t = await createTestSession({ extensions: [goalPath()] });
174 278 installSaysStream(t, ["verified files and tests"]);
176 280 await t.session.prompt("/goal finish feature");
177 281 await waitFor(() => latestGoal()?.status === "complete");
179 - expect(latestGoal()).toMatchObject({ objective: "finish feature", status: "complete", lastReason: "all criteria verified" });
283 + expect(latestGoal()).toMatchObject({
284 + objective: "finish feature",
285 + status: "complete",
286 + lastReason: "all criteria verified",
287 + });
180 288 expect(customEvents("goal-judge")).toMatchObject([
181 289 { event: "start", objective: "finish feature", turn: 1, maxTurns: 20 },
182 - { event: "verdict", objective: "finish feature", turn: 1, maxTurns: 20, verdict: { done: true, blocked: false, reason: "all criteria verified" } },
291 + event: "verdict",
292 + objective: "finish feature",
293 + turn: 1,
294 + maxTurns: 20,
295 + verdict: {
296 + done: true,
297 + blocked: false,
298 + reason: "all criteria verified",
299 + },
300 + },
184 - const statusCalls = t.events.ui.filter((call) => call.method === "setStatus" && call.args[0] === "goal").map((call) => call.args[1]);
302 + const statusCalls = t.events.ui
303 + .filter(
304 + (call) => call.method === "setStatus" && call.args[0] === "goal",
306 + .map((call) => call.args[1]);
185 307 expect(statusCalls).toContain("goal judging 1/20");
186 - const widgetCalls = t.events.ui.filter((call) => call.method === "setWidget" && call.args[0] === "goal").map((call) => call.args[1]);
308 + const widgetCalls = t.events.ui
309 + .filter(
310 + (call) => call.method === "setWidget" && call.args[0] === "goal",
312 + .map((call) => call.args[1]);
187 313 expect(widgetCalls.some(Boolean)).toBe(true);
188 314 expect(widgetCalls).toContain(undefined);
189 315 } finally {
320 + it("persists visible judge transcript rows without another agent turn", async () => {
321 + const restore = installJudgeProvider([
322 + '{"done":true,"blocked":false,"reason":"visible verdict"}',
323 + ]);
324 + try {
325 + t = await createTestSession({ extensions: [goalPath()] });
326 + installSaysStream(t, ["verified"]);
328 + await t.session.prompt("/goal show judge rows");
329 + await waitFor(() => customMessages("goal-judge").length >= 2);
331 + expect(customMessages("goal-judge")).toMatchObject([
333 + content: "goal judge running · turn 1/20",
334 + display: true,
335 + details: { event: "start", turn: 1, maxTurns: 20 },
336 + },
338 + content: "goal judge done · turn 1/20 · visible verdict",
339 + display: true,
340 + details: { event: "verdict", turn: 1, maxTurns: 20 },
341 + },
342 + ]);
343 + expect(
344 + t.events.all.filter((event) => event.type === "agent_start"),
345 + ).toHaveLength(1);
346 + } finally {
347 + restore();
349 + });
351 + it("retries an empty judge response", async () => {
352 + const restore = installJudgeProvider([
353 + "",
354 + '{"done":true,"blocked":false,"reason":"retry succeeded"}',
355 + ]);
356 + try {
357 + t = await createTestSession({ extensions: [goalPath()] });
358 + installSaysStream(t, ["verified after retry"]);
360 + await t.session.prompt("/goal retry judge");
361 + await waitFor(() => latestGoal()?.status === "complete");
363 + expect(latestGoal()).toMatchObject({
364 + objective: "retry judge",
365 + status: "complete",
366 + lastReason: "retry succeeded",
367 + });
368 + } finally {
369 + restore();
371 + });
194 373 it("judge blocked verdict blocks goal", async () => {
195 - const restore = installJudgeProvider(['{"done":false,"blocked":true,"reason":"needs credentials"}']);
374 + const restore = installJudgeProvider([
375 + '{"done":false,"blocked":true,"reason":"needs credentials"}',
376 + ]);
197 378 t = await createTestSession({ extensions: [goalPath()] });
198 379 installSaysStream(t, ["cannot continue without credentials"]);
200 381 await t.session.prompt("/goal deploy service");
201 382 await waitFor(() => latestGoal()?.status === "blocked");
203 - expect(latestGoal()).toMatchObject({ objective: "deploy service", status: "blocked", lastReason: "needs credentials" });
384 + expect(latestGoal()).toMatchObject({
385 + objective: "deploy service",
386 + status: "blocked",
387 + lastReason: "needs credentials",
388 + });
204 389 } finally {
205 390 restore();
216 401 await t.session.prompt("/goal finish migration");
217 402 await waitFor(() => latestGoal()?.status === "paused", 2_000);
219 - expect(latestGoal()).toMatchObject({ objective: "finish migration", status: "paused", turnsUsed: 20, lastReason: "turn budget exhausted: needs more work" });
404 + expect(latestGoal()).toMatchObject({
405 + objective: "finish migration",
406 + status: "paused",
407 + turnsUsed: 20,
408 + lastReason: "turn budget exhausted: needs more work",
409 + });
220 410 } finally {
221 411 restore();
236 426 for (const mode of ["direct", "microtask"] as const) {
237 427 it(`${mode} dispatch is still busy inside agent_end`, async () => {
238 428 await runProbe(mode, 1);
239 - const agentStarts = t!.events.all.filter((event) => event.type === "agent_start");
240 - const agentEnds = t!.events.all.filter((event) => event.type === "agent_end");
241 - const probeEvents = customEvents("goal-probe").map((entry) => entry.event);
429 + const agentStarts = t!.events.all.filter(
430 + (event) => event.type === "agent_start",
431 + );
432 + const agentEnds = t!.events.all.filter(
433 + (event) => event.type === "agent_end",
434 + );
435 + const probeEvents = customEvents("goal-probe").map(
436 + (entry) => entry.event,
437 + );
243 439 expect(agentStarts).toHaveLength(1);
244 440 expect(agentEnds).toHaveLength(1);
250 446 it("setTimeout(0) schedules exactly one continuation after idle", async () => {
251 447 await runProbe("timeout", 2);
253 - const agentStarts = t!.events.all.filter((event) => event.type === "agent_start");
254 - const agentEnds = t!.events.all.filter((event) => event.type === "agent_end");
449 + const agentStarts = t!.events.all.filter(
450 + (event) => event.type === "agent_start",
451 + );
452 + const agentEnds = t!.events.all.filter(
453 + (event) => event.type === "agent_end",
454 + );
255 455 const assistantMessages = t!.events.messages
256 456 .filter((message) => message.role === "assistant")
257 457 .map((message) =>
258 458 Array.isArray(message.content)
259 - ? message.content.map((part: { type?: string; text?: string }) => part.type === "text" ? part.text : "").join("\n")
459 + ? message.content
460 + .map((part: { type?: string; text?: string }) =>
461 + part.type === "text" ? part.text : "",
463 + .join("\n")
262 466 const probeEvents = customEvents("goal-probe").map((entry) => entry.event);
265 469 expect(agentEnds).toHaveLength(2);
266 470 expect(assistantMessages).toEqual(["first turn", "second turn"]);
267 471 expect(probeEvents.filter((event) => event === "dispatch")).toHaveLength(1);
268 - expect(probeEvents.indexOf("persisted-before-schedule")).toBeLessThan(probeEvents.indexOf("dispatch"));
472 + expect(probeEvents.indexOf("persisted-before-schedule")).toBeLessThan(
473 + probeEvents.indexOf("dispatch"),
474 + );
269 475 expect(probeEvents.at(-1)).toBe("done");
271 - const userMessages = t!.session.sessionManager.getEntries()
272 - .filter((entry: { type: string; message?: { role?: string; content?: Array<{ type?: string; text?: string }> } }) => entry.type === "message" && entry.message?.role === "user")
273 - .map((entry: { message?: { content?: Array<{ type?: string; text?: string }> } }) => entry.message?.content?.map((part) => part.text ?? "").join("\n"));
477 + const userMessages = t!.session.sessionManager
478 + .getEntries()
479 + .filter(
480 + (entry: {
481 + type: string;
482 + message?: {
483 + role?: string;
484 + content?: Array<{ type?: string; text?: string }>;
485 + };
486 + }) => entry.type === "message" && entry.message?.role === "user",
488 + .map(
489 + (entry: {
490 + message?: { content?: Array<{ type?: string; text?: string }> };
491 + }) => entry.message?.content?.map((part) => part.text ?? "").join("\n"),
492 + );
274 493 expect(userMessages).toContain(GOAL_PROBE_MESSAGE);
282 501 type: "custom",
283 502 customType: "goal-state",
284 - data: { id: "g", objective: "do it", criteria: [], status: "active", turnsUsed: 0, maxTurns: 20, createdAt: now, updatedAt: now },
503 + data: {
504 + id: "g",
505 + objective: "do it",
506 + criteria: [],
507 + status: "active",
508 + turnsUsed: 0,
509 + maxTurns: 20,
510 + createdAt: now,
511 + updatedAt: now,
512 + },
513 + },
515 + type: "custom",
516 + customType: "goal-state",
517 + data: { cleared: true, updatedAt: now + 1 },
286 - { type: "custom", customType: "goal-state", data: { cleared: true, updatedAt: now + 1 } },
289 - expect(__goalTest.latestGoalState({ sessionManager: { getEntries: () => entries } } as never)).toBeUndefined();
521 + expect(
522 + __goalTest.latestGoalState({
523 + sessionManager: { getEntries: () => entries },
524 + } as never),
525 + ).toBeUndefined();
292 528 it("strictly parses judge booleans", () => {
293 - expect(__goalTest.parseJudge('{"done":"false","blocked":"false","reason":"no"}')).toEqual({ done: false, blocked: false, reason: "no" });
294 - expect(__goalTest.parseJudge('{"done":true,"blocked":false,"reason":"yes"}')).toEqual({ done: true, blocked: false, reason: "yes" });
529 + expect(
530 + __goalTest.parseJudge('{"done":"false","blocked":"false","reason":"no"}'),
531 + ).toEqual({ done: false, blocked: false, reason: "no" });
532 + expect(
533 + __goalTest.parseJudge('{"done":true,"blocked":false,"reason":"yes"}'),
534 + ).toEqual({ done: true, blocked: false, reason: "yes" });
297 537 it("ignores stale judge result after goal is cleared during await", async () => {
298 538 const previousProvider = getApiProvider("openai-responses");
299 539 let resolveJudge: ((text: string) => void) | undefined;
300 540 let judgeCalled = false;
301 - registerApiProvider({
302 - api: "openai-responses",
303 - stream: () => {
304 - judgeCalled = true;
305 - const stream = createAssistantMessageEventStream();
306 - new Promise<string>((resolve) => {
307 - resolveJudge = resolve;
308 - }).then((text) => stream.push(doneEvent(text)));
309 - return stream;
541 + registerApiProvider(
543 + api: "openai-responses",
544 + stream: () => {
545 + judgeCalled = true;
546 + const stream = createAssistantMessageEventStream();
547 + new Promise<string>((resolve) => {
548 + resolveJudge = resolve;
549 + }).then((text) => stream.push(doneEvent(text)));
550 + return stream;
551 + },
552 + streamSimple: () => {
553 + throw new Error("not used");
554 + },
311 - streamSimple: () => {
312 - throw new Error("not used");
313 - },
314 - }, "goal-test");
556 + "goal-test",
557 + );
317 560 t = await createTestSession({ extensions: [goalPath()] });
324 567 await t.session.agent.waitForIdle();
325 568 await new Promise((resolve) => setTimeout(resolve, 10));
327 - expect(customEvents("goal-state").map((entry) => entry.status)).not.toContain("complete");
570 + expect(
571 + customEvents("goal-state").map((entry) => entry.status),
572 + ).not.toContain("complete");
328 573 expect(latestGoal()).toBeUndefined();
329 574 } finally {
330 - if (previousProvider) registerApiProvider(previousProvider, "goal-test-restore");
575 + if (previousProvider)
576 + registerApiProvider(previousProvider, "goal-test-restore");

extensions/goal/__tests__/unit.test.ts

Modified
20 20 function replay(data: unknown[]) {
21 - const entries = data.map((entry) => ({ type: "custom", customType: GOAL_STATE_CUSTOM_TYPE, data: entry }));
22 - return __goalTest.latestGoalState({ sessionManager: { getEntries: () => entries } } as never);
21 + const entries = data.map((entry) => ({
22 + type: "custom",
23 + customType: GOAL_STATE_CUSTOM_TYPE,
24 + data: entry,
25 + }));
26 + return __goalTest.latestGoalState({
27 + sessionManager: { getEntries: () => entries },
28 + } as never);
25 31 describe("goal unit helpers", () => {
26 32 it("extracts text content from strings and text parts", () => {
27 33 expect(__goalTest.textFromContent("plain")).toBe("plain");
28 - expect(__goalTest.textFromContent([{ type: "text", text: "a" }, { type: "image", url: "x" }, { type: "text", text: "b" }])).toBe("a\nb");
34 + expect(
35 + __goalTest.textFromContent([
36 + { type: "text", text: "a" },
37 + { type: "image", url: "x" },
38 + { type: "text", text: "b" },
39 + ]),
40 + ).toBe("a\nb");
29 41 expect(__goalTest.textFromContent({ nope: true })).toBe("");
32 44 it("loads settings with clamped integers and trimmed judge model", () => {
33 - expect(__goalTest.loadSettings({ settingsManager: { getSettings: () => ({ goal: { maxTurns: "3.9", tailBytes: 1_000_000, judgeModel: " test/model " } }) } } as never)).toEqual({
45 + expect(
46 + __goalTest.loadSettings({
47 + settingsManager: {
48 + getSettings: () => ({
49 + goal: {
50 + maxTurns: "3.9",
51 + tailBytes: 1_000_000,
52 + judgeModel: " test/model ",
53 + },
54 + }),
55 + },
56 + } as never),
57 + ).toEqual({
34 58 maxTurns: 3,
35 59 judgeModel: "test/model",
36 60 tailBytes: 65_536,
38 - expect(__goalTest.loadSettings({ settingsManager: { getSettings: () => ({ goal: { maxTurns: -1, tailBytes: "bad", judgeModel: " " } }) } } as never)).toEqual({
62 + expect(
63 + __goalTest.loadSettings({
64 + settingsManager: {
65 + getSettings: () => ({
66 + goal: { maxTurns: -1, tailBytes: "bad", judgeModel: " " },
67 + }),
68 + },
69 + } as never),
70 + ).toEqual({
39 71 maxTurns: 1,
40 72 judgeModel: undefined,
41 73 tailBytes: 4096,
45 77 it("parses model refs and rejects invalid refs", () => {
46 - expect(__goalTest.parseModelRef("provider/model/name")).toEqual({ provider: "provider", model: "model/name" });
47 - expect(() => __goalTest.parseModelRef("missing-slash")).toThrow('Invalid model ref "missing-slash". Use "provider/model".');
48 - expect(() => __goalTest.parseModelRef("provider/")).toThrow('Invalid model ref "provider/". Use "provider/model".');
78 + expect(__goalTest.parseModelRef("provider/model/name")).toEqual({
79 + provider: "provider",
80 + model: "model/name",
81 + });
82 + expect(() => __goalTest.parseModelRef("missing-slash")).toThrow(
83 + 'Invalid model ref "missing-slash". Use "provider/model".',
84 + );
85 + expect(() => __goalTest.parseModelRef("provider/")).toThrow(
86 + 'Invalid model ref "provider/". Use "provider/model".',
87 + );
51 90 it("extracts standing goal objective blocks", () => {
52 - expect(__goalTest.extractStandingGoalObjective("Start working toward this standing goal.\n\n<objective>\nShip it\n</objective>")).toBe("Ship it");
53 - expect(__goalTest.extractStandingGoalObjective("Continue working toward this standing goal.\n\n<objective>\n Ship it\n now\n</objective>")).toBe("Ship it\n now");
54 - expect(__goalTest.extractStandingGoalObjective("<objective>\nShip it\n</objective>")).toBeUndefined();
55 - expect(__goalTest.extractStandingGoalObjective("standing goal without xml")).toBeUndefined();
91 + expect(
92 + __goalTest.extractStandingGoalObjective(
93 + "Start working toward this standing goal.\n\n<objective>\nShip it\n</objective>",
94 + ),
95 + ).toBe("Ship it");
96 + expect(
97 + __goalTest.extractStandingGoalObjective(
98 + "Continue working toward this standing goal.\n\n<objective>\n Ship it\n now\n</objective>",
99 + ),
100 + ).toBe("Ship it\n now");
101 + expect(
102 + __goalTest.extractStandingGoalObjective(
103 + "<objective>\nShip it\n</objective>",
104 + ),
105 + ).toBeUndefined();
106 + expect(
107 + __goalTest.extractStandingGoalObjective("standing goal without xml"),
108 + ).toBeUndefined();
109 + });
111 + it("formats judge reasons as bounded one-line text", () => {
112 + expect(__goalTest.judgeDisplayReason(" a\n\t b ")).toBe("a b");
113 + const reason = __goalTest.judgeDisplayReason("x".repeat(200));
114 + expect(reason).toHaveLength(160);
115 + expect(reason.endsWith("…")).toBe(true);
58 118 it("parses judge JSON strictly", () => {
59 - expect(__goalTest.parseJudge('prefix {"done":true,"blocked":false,"reason":"complete"} suffix')).toEqual({ done: true, blocked: false, reason: "complete" });
60 - expect(__goalTest.parseJudge('{"done":"true","blocked":1,"reason":" loose "}')).toEqual({ done: false, blocked: false, reason: "loose" });
61 - expect(__goalTest.parseJudge('{"done":false,"blocked":true}')).toEqual({ done: false, blocked: true, reason: "no reason" });
62 - expect(() => __goalTest.parseJudge("[]")).toThrow("judge returned non-object JSON");
119 + expect(
120 + __goalTest.parseJudge(
121 + 'prefix {"done":true,"blocked":false,"reason":"complete"} suffix',
122 + ),
123 + ).toEqual({ done: true, blocked: false, reason: "complete" });
124 + expect(
125 + __goalTest.parseJudge('{"done":"true","blocked":1,"reason":" loose "}'),
126 + ).toEqual({ done: false, blocked: false, reason: "loose" });
127 + expect(__goalTest.parseJudge('{"done":false,"blocked":true}')).toEqual({
128 + done: false,
129 + blocked: true,
130 + reason: "no reason",
131 + });
132 + expect(() => __goalTest.parseJudge("[]")).toThrow(
133 + "judge returned non-object JSON",
134 + );
135 + expect(() => __goalTest.parseJudge("")).toThrow(
136 + "judge returned empty response",
137 + );
63 138 expect(() => __goalTest.parseJudge("not json")).toThrow();
66 141 it("replays newest valid goal state, tombstones, and malformed entries", () => {
67 - expect(replay([goal({ id: "old" }), goal({ id: "new", turnsUsed: 4 })])?.id).toBe("new");
68 - expect(replay([goal(), { cleared: true, updatedAt: now + 1 }])).toBeUndefined();
69 - expect(replay([{ id: "bad", objective: "missing fields" }, goal({ id: "good" })])?.id).toBe("good");
70 - expect(replay([goal({ id: "old" }), { id: "bad", objective: "missing fields" }])?.id).toBe("old");
142 + expect(
143 + replay([goal({ id: "old" }), goal({ id: "new", turnsUsed: 4 })])?.id,
144 + ).toBe("new");
145 + expect(
146 + replay([goal(), { cleared: true, updatedAt: now + 1 }]),
147 + ).toBeUndefined();
148 + expect(
149 + replay([{ id: "bad", objective: "missing fields" }, goal({ id: "good" })])
150 + ?.id,
151 + ).toBe("good");
152 + expect(
153 + replay([goal({ id: "old" }), { id: "bad", objective: "missing fields" }])
154 + ?.id,
155 + ).toBe("old");
73 158 it("reads latest assistant tail", () => {
74 159 const entries = [
75 - { type: "message", message: { role: "assistant", content: [{ type: "text", text: "first" }] } },
76 - { type: "message", message: { role: "user", content: [{ type: "text", text: "ignored" }] } },
77 - { type: "message", message: { role: "assistant", content: [{ type: "text", text: "abcdef" }] } },
161 + type: "message",
162 + message: {
163 + role: "assistant",
164 + content: [{ type: "text", text: "first" }],
165 + },
166 + },
168 + type: "message",
169 + message: { role: "user", content: [{ type: "text", text: "ignored" }] },
170 + },
172 + type: "message",
173 + message: {
174 + role: "assistant",
175 + content: [{ type: "text", text: "abcdef" }],
176 + },
177 + },
79 - expect(__goalTest.latestAssistantTail({ sessionManager: { getEntries: () => entries } } as never, 3)).toBe("def");
179 + expect(
180 + __goalTest.latestAssistantTail(
181 + { sessionManager: { getEntries: () => entries } } as never,
182 + 3,
183 + ),
184 + ).toBe("def");
82 187 it("autocompletes goal subcommands and criterion removal", () => {
83 - expect(__goalTest.goalArgumentCompletions("pa", undefined)).toEqual([{ value: "pause", label: "pause", description: "Pause active goal" }]);
84 - expect(__goalTest.goalArgumentCompletions("clear", undefined)?.map((item) => item.value)).toEqual(["clear", "clear-criteria"]);
85 - expect(__goalTest.goalArgumentCompletions("remove ", goal() as never)).toEqual([
188 + expect(__goalTest.goalArgumentCompletions("pa", undefined)).toEqual([
189 + { value: "pause", label: "pause", description: "Pause active goal" },
190 + ]);
191 + expect(
192 + __goalTest
193 + .goalArgumentCompletions("clear", undefined)
194 + ?.map((item) => item.value),
195 + ).toEqual(["clear", "clear-criteria"]);
196 + expect(
197 + __goalTest.goalArgumentCompletions("remove ", goal() as never),
198 + ).toEqual([
86 199 { value: "remove 1", label: "1", description: "tests pass" },
87 200 { value: "remove 2", label: "2", description: "docs updated" },
89 - expect(__goalTest.goalArgumentCompletions("remove 2", goal() as never)).toEqual([{ value: "remove 2", label: "2", description: "docs updated" }]);
90 - expect(__goalTest.goalArgumentCompletions("add ", goal() as never)).toBeNull();
202 + expect(
203 + __goalTest.goalArgumentCompletions("remove 2", goal() as never),
204 + ).toEqual([{ value: "remove 2", label: "2", description: "docs updated" }]);
205 + expect(
206 + __goalTest.goalArgumentCompletions("add ", goal() as never),
207 + ).toBeNull();
93 210 it("marks goals owned by another session in status output", () => {
94 - expect(__goalTest.statusLine(goal({ sessionId: "other" }) as never, "current")).toContain("owned by another session");
95 - expect(__goalTest.statusLine(goal({ sessionId: "current" }) as never, "current")).not.toContain("owned by another session");
211 + expect(
212 + __goalTest.statusLine(goal({ sessionId: "other" }) as never, "current"),
213 + ).toContain("owned by another session");
214 + expect(
215 + __goalTest.statusLine(goal({ sessionId: "current" }) as never, "current"),
216 + ).not.toContain("owned by another session");
98 219 it("hides turn budget for terminal goal statuses", () => {
99 - expect(__goalTest.statusText(goal({ status: "active" }) as never)).toBe("goal active 2/5 · 2c");
100 - expect(__goalTest.statusText(goal({ status: "paused" }) as never)).toBe("goal paused 2/5 · 2c");
101 - expect(__goalTest.statusText(goal({ status: "complete", turnsUsed: 0 }) as never)).toBeUndefined();
102 - expect(__goalTest.statusText(goal({ status: "blocked", turnsUsed: 0 }) as never)).toBe("goal blocked");
103 - expect(__goalTest.statusLine(goal({ status: "complete", turnsUsed: 0 }) as never)).not.toContain("0/5");
220 + expect(__goalTest.statusText(goal({ status: "active" }) as never)).toBe(
221 + "goal active 2/5 · 2c",
222 + );
223 + expect(__goalTest.statusText(goal({ status: "paused" }) as never)).toBe(
224 + "goal paused 2/5 · 2c",
225 + );
226 + expect(
227 + __goalTest.statusText(
228 + goal({ status: "complete", turnsUsed: 0 }) as never,
229 + ),
230 + ).toBeUndefined();
231 + expect(
232 + __goalTest.statusText(goal({ status: "blocked", turnsUsed: 0 }) as never),
233 + ).toBe("goal blocked");
234 + expect(
235 + __goalTest.statusLine(
236 + goal({ status: "complete", turnsUsed: 0 }) as never,
237 + ),
238 + ).not.toContain("0/5");
106 241 it("shows useful judge progress context", () => {
107 - expect(__goalTest.judgeRunningText(goal({ turnsUsed: 2, maxTurns: 5 }) as never)).toBe("Goal judge running: turn 3/5 · 2 criteria · Ship the feature");
108 - expect(__goalTest.judgeRunningText(goal({ objective: "x".repeat(100), criteria: [] }) as never)).toBe(`Goal judge running: turn 3/5 · ${"x".repeat(71)}…`);
242 + expect(
243 + __goalTest.judgeRunningText(goal({ turnsUsed: 2, maxTurns: 5 }) as never),
244 + ).toBe("Goal judge running: turn 3/5 · 2 criteria · Ship the feature");
245 + expect(
246 + __goalTest.judgeRunningText(
247 + goal({ objective: "x".repeat(100), criteria: [] }) as never,
248 + ),
249 + ).toBe(`Goal judge running: turn 3/5 · ${"x".repeat(71)}…`);
111 252 it("snapshots user-facing prompt/status output", () => {
112 253 const state = goal({ lastReason: "still checking" }) as never;
113 - expect(JSON.stringify({
114 - status: __goalTest.statusLine(state),
115 - criteria: __goalTest.criteriaText(state),
116 - active: __goalTest.activeGoalPrompt(state),
117 - continuation: __goalTest.continuationPrompt(state),
118 - kickoff: __goalTest.kickoffPrompt(state),
119 - }, null, 2)).toMatchInlineSnapshot(`
254 + expect(
255 + JSON.stringify(
257 + status: __goalTest.statusLine(state),
258 + criteria: __goalTest.criteriaText(state),
259 + active: __goalTest.activeGoalPrompt(state),
260 + continuation: __goalTest.continuationPrompt(state),
261 + kickoff: __goalTest.kickoffPrompt(state),
262 + },
263 + null,
264 + 2,
265 + ),
266 + ).toMatchInlineSnapshot(`
121 268 "status": "active · 2/5 turns · 2 criteria · still checking\\nShip the feature",
122 269 "criteria": "1. tests pass\\n2. docs updated",

extensions/goal/bench/goal-state.bench.ts

Modified
1 1 import { __goalTest, GOAL_STATE_CUSTOM_TYPE } from "../index.ts";
3 -type Entry = { type: string; customType?: string; data?: unknown; message?: unknown };
3 +type Entry = {
4 + type: string;
5 + customType?: string;
6 + data?: unknown;
7 + message?: unknown;
8 +};
4 9 type BenchCtx = { sessionManager: { getEntries(): Entry[] } };
6 11 const now = 1_700_000_000_000;
18 23 function message(i: number): Entry {
19 - return { type: "message", message: { role: i % 2 === 0 ? "user" : "assistant", content: [{ type: "text", text: `message ${i}` }] } };
24 + return {
25 + type: "message",
26 + message: {
27 + role: i % 2 === 0 ? "user" : "assistant",
28 + content: [{ type: "text", text: `message ${i}` }],
29 + },
30 + };
22 -function makeEntries(kind: "latest-goal" | "cleared" | "none", count: number): Entry[] {
33 +function makeEntries(
34 + kind: "latest-goal" | "cleared" | "none",
35 + count: number,
36 +): Entry[] {
23 37 const entries = Array.from({ length: count }, (_, i) => message(i));
24 - if (kind === "latest-goal") entries.push({ type: "custom", customType: GOAL_STATE_CUSTOM_TYPE, data: goal });
38 + if (kind === "latest-goal")
39 + entries.push({
40 + type: "custom",
41 + customType: GOAL_STATE_CUSTOM_TYPE,
42 + data: goal,
43 + });
25 44 if (kind === "cleared") {
26 - entries.push({ type: "custom", customType: GOAL_STATE_CUSTOM_TYPE, data: goal });
27 - entries.push({ type: "custom", customType: GOAL_STATE_CUSTOM_TYPE, data: { cleared: true, updatedAt: now + 1 } });
45 + entries.push({
46 + type: "custom",
47 + customType: GOAL_STATE_CUSTOM_TYPE,
48 + data: goal,
49 + });
50 + entries.push({
51 + type: "custom",
52 + customType: GOAL_STATE_CUSTOM_TYPE,
53 + data: { cleared: true, updatedAt: now + 1 },
54 + });
29 56 return entries;
40 67 function isGoalStatus(value: unknown): boolean {
41 - return value === "active" || value === "paused" || value === "complete" || value === "blocked";
68 + return (
69 + value === "active" ||
70 + value === "paused" ||
71 + value === "complete" ||
72 + value === "blocked"
73 + );
44 76 function isGoalState(value: unknown): boolean {
45 77 if (!isRecord(value)) return false;
46 - return typeof value.id === "string"
47 - && typeof value.objective === "string"
48 - && Array.isArray(value.criteria)
49 - && value.criteria.every((criterion) => typeof criterion === "string")
50 - && isGoalStatus(value.status)
51 - && Number.isInteger(value.turnsUsed)
52 - && Number.isInteger(value.maxTurns)
53 - && typeof value.createdAt === "number"
54 - && typeof value.updatedAt === "number";
78 + return (
79 + typeof value.id === "string" &&
80 + typeof value.objective === "string" &&
81 + Array.isArray(value.criteria) &&
82 + value.criteria.every((criterion) => typeof criterion === "string") &&
83 + isGoalStatus(value.status) &&
84 + Number.isInteger(value.turnsUsed) &&
85 + Number.isInteger(value.maxTurns) &&
86 + typeof value.createdAt === "number" &&
87 + typeof value.updatedAt === "number"
88 + );
57 91 function latestGoalStateForwardBaseline(c: BenchCtx): unknown {
58 92 let state: unknown;
59 93 for (const entry of c.sessionManager.getEntries()) {
60 - if (entry.type !== "custom" || entry.customType !== GOAL_STATE_CUSTOM_TYPE || !isRecord(entry.data)) continue;
94 + if (
95 + entry.type !== "custom" ||
96 + entry.customType !== GOAL_STATE_CUSTOM_TYPE ||
97 + !isRecord(entry.data)
98 + )
99 + continue;
61 100 if (entry.data.cleared === true) {
62 101 state = undefined;
63 102 continue;
67 106 return state;
70 -const latestGoalState = process.env.GOAL_BENCH_IMPL === "forward"
71 - ? latestGoalStateForwardBaseline
72 - : (c: BenchCtx) => __goalTest.latestGoalState(c as never);
109 +const latestGoalState =
110 + process.env.GOAL_BENCH_IMPL === "forward"
111 + ? latestGoalStateForwardBaseline
112 + : (c: BenchCtx) => __goalTest.latestGoalState(c as never);
74 114 let sink: unknown;
84 124 if (elapsed < minMs * 1_000_000) n *= 2;
86 126 const nsPerOp = elapsed / n;
87 - console.log(`Benchmark${name}-${process.arch}\t${n}\t${nsPerOp.toFixed(0)} ns/op`);
127 + console.log(
128 + `Benchmark${name}-${process.arch}\t${n}\t${nsPerOp.toFixed(0)} ns/op`,
129 + );
90 132 const size = Number(process.env.GOAL_BENCH_ENTRIES ?? 100_000);

extensions/goal/index.ts

Modified
1 1 import { complete, StringEnum } from "@earendil-works/pi-ai";
2 -import type { ExtensionAPI, ExtensionContext, ThemeColor } from "@earendil-works/pi-coding-agent";
3 -import { truncateToWidth, type AutocompleteItem } from "@earendil-works/pi-tui";
2 +import type {
3 + ExtensionAPI,
4 + ExtensionContext,
5 + ThemeColor,
6 +} from "@earendil-works/pi-coding-agent";
7 +import {
8 + type AutocompleteItem,
9 + Text,
10 + truncateToWidth,
11 +} from "@earendil-works/pi-tui";
4 12 import { Type } from "typebox";
6 14 type GoalStatus = "active" | "paused" | "complete" | "blocked";
55 63 const FOOTER_SEGMENT = { id: "goal", zone: "llm", order: 30 } as const;
56 64 const CONTINUATION_RETRY_MS = 250;
57 65 const CONTINUATION_MAX_ATTEMPTS = 20;
66 +const JUDGE_ATTEMPTS = 2;
58 67 const JUDGE_SYSTEM_PROMPT = `You are a strict /goal completion judge.
59 68 Return only JSON: {"done":boolean,"blocked":boolean,"reason":"one sentence"}.
60 69 Mark done only if the latest assistant response explicitly proves the objective and every criterion are complete.
82 91 return value !== null && typeof value === "object" && !Array.isArray(value);
85 -function clampInt(value: unknown, fallback: number, min: number, max: number): number {
94 +function clampInt(
95 + value: unknown,
96 + fallback: number,
97 + min: number,
98 + max: number,
99 +): number {
86 100 const n = Number(value ?? fallback);
87 101 if (!Number.isFinite(n)) return fallback;
88 102 return Math.max(min, Math.min(max, Math.trunc(n)));
93 107 const raw = isRecord(all?.goal) ? all.goal : {};
94 108 return {
95 109 maxTurns: clampInt(raw.maxTurns, DEFAULT_MAX_TURNS, 1, 10_000),
96 - judgeModel: typeof raw.judgeModel === "string" && raw.judgeModel.trim() ? raw.judgeModel.trim() : undefined,
110 + judgeModel:
111 + typeof raw.judgeModel === "string" && raw.judgeModel.trim()
112 + ? raw.judgeModel.trim()
113 + : undefined,
97 114 tailBytes: clampInt(raw.tailBytes, DEFAULT_TAIL_BYTES, 512, 65_536),
104 121 return content
105 122 .map((part) => {
106 123 if (!isRecord(part)) return "";
107 - return part.type === "text" && typeof part.text === "string" ? part.text : "";
124 + return part.type === "text" && typeof part.text === "string"
125 + ? part.text
126 + : "";
109 128 .filter(Boolean)
110 129 .join("\n");
120 139 function isGoalStatus(value: unknown): value is GoalStatus {
121 - return value === "active" || value === "paused" || value === "complete" || value === "blocked";
140 + return (
141 + value === "active" ||
142 + value === "paused" ||
143 + value === "complete" ||
144 + value === "blocked"
145 + );
124 148 function isGoalState(value: unknown): value is GoalState {
125 149 if (!isRecord(value)) return false;
126 - return typeof value.id === "string"
127 - && typeof value.objective === "string"
128 - && Array.isArray(value.criteria)
129 - && value.criteria.every((criterion) => typeof criterion === "string")
130 - && isGoalStatus(value.status)
131 - && Number.isInteger(value.turnsUsed)
132 - && Number.isInteger(value.maxTurns)
133 - && (value.sessionId === undefined || typeof value.sessionId === "string")
134 - && typeof value.createdAt === "number"
135 - && typeof value.updatedAt === "number";
150 + return (
151 + typeof value.id === "string" &&
152 + typeof value.objective === "string" &&
153 + Array.isArray(value.criteria) &&
154 + value.criteria.every((criterion) => typeof criterion === "string") &&
155 + isGoalStatus(value.status) &&
156 + Number.isInteger(value.turnsUsed) &&
157 + Number.isInteger(value.maxTurns) &&
158 + (value.sessionId === undefined || typeof value.sessionId === "string") &&
159 + typeof value.createdAt === "number" &&
160 + typeof value.updatedAt === "number"
161 + );
138 164 function latestGoalState(ctx: ExtensionContext): GoalState | undefined {
139 - const entries = ctx.sessionManager.getEntries() as Array<{ type: string; customType?: string; data?: unknown }>;
165 + const entries = ctx.sessionManager.getEntries() as Array<{
166 + type: string;
167 + customType?: string;
168 + data?: unknown;
169 + }>;
140 170 // Session entries are append-only: newest valid goal-state/tombstone decides replay.
141 171 // Scan backward to avoid O(full transcript) work after long sessions.
142 172 for (let i = entries.length - 1; i >= 0; i--) {
143 173 const entry = entries[i];
144 - if (entry.type !== "custom" || entry.customType !== GOAL_STATE_ENTRY || !isRecord(entry.data)) continue;
174 + if (
175 + entry.type !== "custom" ||
176 + entry.customType !== GOAL_STATE_ENTRY ||
177 + !isRecord(entry.data)
179 + continue;
145 180 if (entry.data.cleared === true) return undefined;
146 181 if (isGoalState(entry.data)) return entry.data;
152 187 pi.appendEntry(GOAL_STATE_ENTRY, state);
155 -function updateGoal(pi: ExtensionAPI, current: GoalState, patch: Partial<GoalState>): GoalState {
190 +function updateGoal(
191 + pi: ExtensionAPI,
192 + current: GoalState,
193 + patch: Partial<GoalState>,
194 +): GoalState {
156 195 const next = { ...current, ...patch, updatedAt: Date.now() };
157 196 appendGoal(pi, next);
158 197 return next;
171 210 return oneLine.length <= max ? oneLine : `${oneLine.slice(0, max - 1)}…`;
213 +function judgeDisplayReason(reason: string): string {
214 + return shortText(reason, 160);
217 +function judgeVerdictLabel(
218 + verdict: JudgeVerdict,
219 +): "done" | "blocked" | "continue" {
220 + if (verdict.done) return "done";
221 + if (verdict.blocked) return "blocked";
222 + return "continue";
174 225 function judgeRunningText(state: GoalState): string {
175 - const criteria = state.criteria.length === 0 ? "" : ` · ${state.criteria.length} criteria`;
226 + const criteria =
227 + state.criteria.length === 0 ? "" : ` · ${state.criteria.length} criteria`;
176 228 return `Goal judge running: turn ${state.turnsUsed + 1}/${state.maxTurns}${criteria} · ${shortText(state.objective)}`;
179 231 function statusLine(state: GoalState, currentSessionId?: string): string {
180 - const turns = showsTurnBudget(state) ? ` · ${state.turnsUsed}/${state.maxTurns} turns` : "";
181 - const criteria = state.criteria.length === 0 ? "" : ` · ${state.criteria.length} criteria`;
232 + const turns = showsTurnBudget(state)
233 + ? ` · ${state.turnsUsed}/${state.maxTurns} turns`
234 + : "";
235 + const criteria =
236 + state.criteria.length === 0 ? "" : ` · ${state.criteria.length} criteria`;
182 237 const reason = state.lastReason ? ` · ${state.lastReason}` : "";
183 - const owner = state.sessionId && currentSessionId && state.sessionId !== currentSessionId ? " · owned by another session" : "";
238 + const owner =
239 + state.sessionId && currentSessionId && state.sessionId !== currentSessionId
240 + ? " · owned by another session"
241 + : "";
184 242 return `${state.status}${turns}${criteria}${owner}${reason}\n${state.objective}`;
187 245 function statusText(state: GoalState | undefined): string | undefined {
188 246 if (!state || !showsPersistentStatus(state)) return undefined;
189 247 if (!showsTurnBudget(state)) return `goal ${state.status}`;
190 - const criteria = state.criteria.length === 0 ? "" : ` · ${state.criteria.length}c`;
248 + const criteria =
249 + state.criteria.length === 0 ? "" : ` · ${state.criteria.length}c`;
191 250 return `goal ${state.status} ${state.turnsUsed}/${state.maxTurns}${criteria}`;
194 253 function judgeStatusText(state: GoalState): string {
195 - const criteria = state.criteria.length === 0 ? "" : ` · ${state.criteria.length}c`;
254 + const criteria =
255 + state.criteria.length === 0 ? "" : ` · ${state.criteria.length}c`;
196 256 return `goal judging ${state.turnsUsed + 1}/${state.maxTurns}${criteria}`;
224 284 function goalProgress(state: GoalState): number {
225 - return state.maxTurns <= 0 ? 0 : Math.max(0, Math.min(100, Math.round((state.turnsUsed / state.maxTurns) * 100)));
285 + return state.maxTurns <= 0
286 + ? 0
287 + : Math.max(
288 + 0,
289 + Math.min(100, Math.round((state.turnsUsed / state.maxTurns) * 100)),
290 + );
228 293 function goalBar(state: GoalState, width = 10): string {
229 - const filled = Math.max(0, Math.min(width, Math.round((goalProgress(state) / 100) * width)));
294 + const filled = Math.max(
295 + 0,
296 + Math.min(width, Math.round((goalProgress(state) / 100) * width)),
297 + );
230 298 return "█".repeat(filled) + "░".repeat(width - filled);
233 301 function goalPhaseText(state: GoalState, mode: GoalUiMode): string {
234 - if (mode === "judging") return `judging ${state.turnsUsed + 1}/${state.maxTurns}`;
302 + if (mode === "judging")
303 + return `judging ${state.turnsUsed + 1}/${state.maxTurns}`;
235 304 if (!showsTurnBudget(state)) return state.status;
236 305 return `${state.status} ${state.turnsUsed}/${state.maxTurns}`;
241 310 return state;
244 -function updateWidget(ctx: ExtensionContext, state: GoalState | undefined, mode: GoalUiMode): void {
313 +function updateWidget(
314 + ctx: ExtensionContext,
315 + state: GoalState | undefined,
316 + mode: GoalUiMode,
317 +): void {
245 318 const shown = widgetState(state);
246 319 if (!shown) {
247 320 ctx.ui.setWidget(WIDGET_KEY, undefined);
248 321 return;
250 - ctx.ui.setWidget(WIDGET_KEY, (_tui, theme) => ({
251 - invalidate() {},
252 - render(width: number): string[] {
253 - const tone = goalTone(shown, mode);
254 - const criteria = shown.criteria.length === 0 ? "" : ` · ${shown.criteria.length}c`;
255 - const turns = showsTurnBudget(shown) ? ` · ${goalBar(shown)} ${goalProgress(shown)}%` : "";
256 - const header = `${theme.fg(tone, goalIcon(shown, mode))} ${theme.fg(tone, theme.bold("goal"))} ${theme.fg(tone, goalPhaseText(shown, mode))}${theme.fg("dim", criteria)}${theme.fg("dim", turns)}`;
257 - const lines = [
258 - header,
259 - theme.fg("dim", "objective: ") + shown.objective,
260 - ];
261 - if (shown.lastReason) lines.push(theme.fg("dim", "judge: ") + shown.lastReason);
262 - return lines.map((line) => truncateToWidth(line, width, "…"));
263 - },
264 - }), { placement: "aboveEditor" });
323 + ctx.ui.setWidget(
324 + WIDGET_KEY,
325 + (_tui, theme) => ({
326 + invalidate() {},
327 + render(width: number): string[] {
328 + const tone = goalTone(shown, mode);
329 + const criteria =
330 + shown.criteria.length === 0 ? "" : ` · ${shown.criteria.length}c`;
331 + const turns = showsTurnBudget(shown)
332 + ? ` · ${goalBar(shown)} ${goalProgress(shown)}%`
333 + : "";
334 + const header = `${theme.fg(tone, goalIcon(shown, mode))} ${theme.fg(tone, theme.bold("goal"))} ${theme.fg(tone, goalPhaseText(shown, mode))}${theme.fg("dim", criteria)}${theme.fg("dim", turns)}`;
335 + const lines = [
336 + header,
337 + theme.fg("dim", "objective: ") + shown.objective,
338 + ];
339 + if (shown.lastReason)
340 + lines.push(theme.fg("dim", "judge: ") + shown.lastReason);
341 + return lines.map((line) => truncateToWidth(line, width, "…"));
342 + },
343 + }),
344 + { placement: "aboveEditor" },
345 + );
267 -function updateFooter(pi: ExtensionAPI, ctx: ExtensionContext, state: GoalState | undefined, mode: GoalUiMode): void {
348 +function updateFooter(
349 + pi: ExtensionAPI,
350 + ctx: ExtensionContext,
351 + state: GoalState | undefined,
352 + mode: GoalUiMode,
353 +): void {
268 354 if (!hasFooter()) {
269 355 clearFooter(pi);
270 - ctx.ui.setStatus(STATUS_KEY, mode === "judging" && state ? judgeStatusText(state) : statusText(state));
356 + ctx.ui.setStatus(
357 + STATUS_KEY,
358 + mode === "judging" && state ? judgeStatusText(state) : statusText(state),
359 + );
271 360 return;
273 362 ctx.ui.setStatus(STATUS_KEY, undefined);
274 363 const shown = widgetState(state);
275 364 if (!shown) return clearFooter(pi);
276 365 const phase = goalPhaseText(shown, mode);
277 - const criteria = shown.criteria.length === 0 ? "" : ` · ${shown.criteria.length}c`;
278 - const reason = shown.lastReason ? ` · ${shortText(shown.lastReason, 36)}` : "";
366 + const criteria =
367 + shown.criteria.length === 0 ? "" : ` · ${shown.criteria.length}c`;
368 + const reason = shown.lastReason
369 + ? ` · ${shortText(shown.lastReason, 36)}`
370 + : "";
279 371 pi.events.emit("footer:segment", {
280 372 ...FOOTER_SEGMENT,
281 373 text: `${phase}${criteria}${reason}`,
287 -function updateUI(pi: ExtensionAPI, ctx: ExtensionContext, state: GoalState | undefined, mode: GoalUiMode = "normal"): void {
379 +function updateUI(
380 + pi: ExtensionAPI,
381 + ctx: ExtensionContext,
382 + state: GoalState | undefined,
383 + mode: GoalUiMode = "normal",
384 +): void {
288 385 if (!ctx.hasUI) return;
289 386 updateWidget(ctx, state, mode);
290 387 updateFooter(pi, ctx, state, mode);
298 395 const names = pi.getActiveTools();
299 396 const has = names.includes(name);
300 397 if (active === has) return;
301 - pi.setActiveTools(active ? [...names, name] : names.filter((tool) => tool !== name));
398 + pi.setActiveTools(
399 + active ? [...names, name] : names.filter((tool) => tool !== name),
400 + );
304 403 function currentSessionId(ctx: ExtensionContext): string | undefined {
309 408 return !state.sessionId || state.sessionId === currentSessionId(ctx);
312 -function ownedGoalOrNotify(ctx: ExtensionContext, state: GoalState | undefined): state is GoalState {
411 +function ownedGoalOrNotify(
412 + ctx: ExtensionContext,
413 + state: GoalState | undefined,
414 +): state is GoalState {
313 415 if (!state) {
314 416 ctx.ui.notify("No goal.", "warning");
315 417 return false;
317 419 if (ownsGoal(ctx, state)) return true;
318 - ctx.ui.notify("Goal belongs to another session — run /goal resume first.", "warning");
420 + ctx.ui.notify(
421 + "Goal belongs to another session — run /goal resume first.",
422 + "warning",
423 + );
319 424 return false;
322 427 function criteriaText(state: GoalState): string {
323 428 if (state.criteria.length === 0) return "No goal criteria.";
324 - return state.criteria.map((criterion, index) => `${index + 1}. ${criterion}`).join("\n");
429 + return state.criteria
430 + .map((criterion, index) => `${index + 1}. ${criterion}`)
431 + .join("\n");
327 -function goalArgumentCompletions(argumentPrefix: string, state: GoalState | undefined): AutocompleteItem[] | null {
434 +function goalArgumentCompletions(
435 + argumentPrefix: string,
436 + state: GoalState | undefined,
437 +): AutocompleteItem[] | null {
328 438 const input = argumentPrefix.trimStart();
329 439 const firstSpace = input.search(/\s/);
330 440 const cmd = firstSpace === -1 ? input : input.slice(0, firstSpace);
331 441 const rest = firstSpace === -1 ? "" : input.slice(firstSpace + 1).trim();
332 442 if (firstSpace === -1) {
333 - const matches = GOAL_SUBCOMMANDS
334 - .filter((subcommand) => subcommand.value.startsWith(cmd))
335 - .map((subcommand) => ({ value: subcommand.value, label: subcommand.value, description: subcommand.description }));
443 + const matches = GOAL_SUBCOMMANDS.filter((subcommand) =>
444 + subcommand.value.startsWith(cmd),
445 + ).map((subcommand) => ({
446 + value: subcommand.value,
447 + label: subcommand.value,
448 + description: subcommand.description,
449 + }));
336 450 return matches.length === 0 ? null : matches;
338 452 if (cmd !== "remove" || !state || state.criteria.length === 0) return null;
339 453 const matches = state.criteria
340 454 .map((criterion, index) => ({ n: String(index + 1), criterion }))
341 455 .filter((item) => item.n.startsWith(rest))
342 - .map((item) => ({ value: `remove ${item.n}`, label: item.n, description: item.criterion }));
456 + .map((item) => ({
457 + value: `remove ${item.n}`,
458 + label: item.n,
459 + description: item.criterion,
460 + }));
343 461 return matches.length === 0 ? null : matches;
346 464 function activeGoalPrompt(state: GoalState): string {
347 - const criteria = state.criteria.length === 0
348 - ? "none"
349 - : state.criteria.map((criterion, index) => `${index + 1}. ${criterion}`).join("\n");
465 + const criteria =
466 + state.criteria.length === 0
467 + ? "none"
468 + : state.criteria
469 + .map((criterion, index) => `${index + 1}. ${criterion}`)
470 + .join("\n");
350 471 return `\n\nActive /goal:\n<objective>\n${state.objective}\n</objective>\n\nCriteria:\n${criteria}\n\nKeep working toward this standing goal. Do not redefine success smaller. Before claiming completion, verify current files, commands, tests, or deliverables. Use update_goal only after judge review and only when complete or genuinely blocked.`;
353 474 function continuationPrompt(state: GoalState): string {
354 - const criteria = state.criteria.length === 0
355 - ? "none"
356 - : state.criteria.map((criterion, index) => `${index + 1}. ${criterion}`).join("\n");
475 + const criteria =
476 + state.criteria.length === 0
477 + ? "none"
478 + : state.criteria
479 + .map((criterion, index) => `${index + 1}. ${criterion}`)
480 + .join("\n");
357 481 return `Continue working toward this standing goal.\n\n<objective>\n${state.objective}\n</objective>\n\nCriteria:\n${criteria}\n\nDo not redefine success smaller. Inspect current state before claiming completion. If incomplete, make concrete progress. If complete or blocked after judge review, call update_goal.`;
364 488 function latestAssistantTail(ctx: ExtensionContext, tailBytes: number): string {
365 - const entries = ctx.sessionManager.getEntries() as Array<{ type: string; message?: { role?: string; content?: unknown } }>;
489 + const entries = ctx.sessionManager.getEntries() as Array<{
490 + type: string;
491 + message?: { role?: string; content?: unknown };
492 + }>;
366 493 for (let i = entries.length - 1; i >= 0; i--) {
367 494 const entry = entries[i];
368 - if (entry?.type !== "message" || entry.message?.role !== "assistant") continue;
495 + if (entry?.type !== "message" || entry.message?.role !== "assistant")
496 + continue;
369 497 const text = textFromContent(entry.message.content).trim();
370 498 if (!text) continue;
371 499 return Buffer.from(text).subarray(-tailBytes).toString();
376 504 function parseModelRef(ref: string): { provider: string; model: string } {
377 505 const i = ref.indexOf("/");
378 - if (i <= 0 || i === ref.length - 1) throw new Error(`Invalid model ref "${ref}". Use "provider/model".`);
506 + if (i <= 0 || i === ref.length - 1)
507 + throw new Error(`Invalid model ref "${ref}". Use "provider/model".`);
379 508 return { provider: ref.slice(0, i), model: ref.slice(i + 1) };
383 512 if (!settings.judgeModel) return ctx.model;
384 513 const { provider, model } = parseModelRef(settings.judgeModel);
385 514 const found = ctx.modelRegistry.find(provider, model);
386 - if (!found) throw new Error(`Goal judge model not found: ${settings.judgeModel}`);
515 + if (!found)
516 + throw new Error(`Goal judge model not found: ${settings.judgeModel}`);
387 517 return found;
390 520 function parseJudge(raw: string): JudgeVerdict {
391 521 const trimmed = raw.trim();
522 + if (!trimmed) throw new Error("judge returned empty response");
392 523 const json = trimmed.match(/\{[\s\S]*\}/)?.[0] ?? trimmed;
393 524 const parsed = JSON.parse(json);
394 525 if (!isRecord(parsed)) throw new Error("judge returned non-object JSON");
395 526 return {
396 527 done: parsed.done === true,
397 528 blocked: parsed.blocked === true,
398 - reason: typeof parsed.reason === "string" && parsed.reason.trim() ? parsed.reason.trim() : "no reason",
529 + reason:
530 + typeof parsed.reason === "string" && parsed.reason.trim()
531 + ? parsed.reason.trim()
532 + : "no reason",
402 -function appendJudge(pi: ExtensionAPI, event: "start" | "verdict", state: GoalState, verdict?: JudgeVerdict): void {
403 - pi.appendEntry(GOAL_JUDGE_ENTRY, {
536 +function appendJudge(
537 + pi: ExtensionAPI,
538 + event: "start" | "verdict",
539 + state: GoalState,
540 + verdict?: JudgeVerdict,
541 +): void {
542 + const data = {
404 543 event,
405 544 goalId: state.id,
406 545 objective: state.objective,
408 547 maxTurns: state.maxTurns,
409 548 verdict,
410 549 timestamp: Date.now(),
411 - });
550 + };
551 + pi.appendEntry(GOAL_JUDGE_ENTRY, data);
552 + setTimeout(() => {
553 + try {
554 + pi.sendMessage(
556 + customType: GOAL_JUDGE_ENTRY,
557 + content: verdict
558 + ? `goal judge ${judgeVerdictLabel(verdict)} · turn ${data.turn}/${data.maxTurns} · ${judgeDisplayReason(verdict.reason)}`
559 + : `goal judge running · turn ${data.turn}/${data.maxTurns}`,
560 + display: true,
561 + details: data,
562 + },
563 + { triggerTurn: false },
564 + );
565 + } catch (error) {
566 + if (!isStaleContextError(error)) throw error;
568 + }, 0);
414 -async function judgeGoal(ctx: ExtensionContext, state: GoalState, settings: GoalSettings): Promise<JudgeVerdict> {
571 +async function judgeGoal(
572 + ctx: ExtensionContext,
573 + state: GoalState,
574 + settings: GoalSettings,
575 +): Promise<JudgeVerdict> {
415 576 const model = resolveJudgeModel(ctx, settings);
416 577 if (!model) return { done: false, blocked: false, reason: "no judge model" };
417 578 const auth = await ctx.modelRegistry.getApiKeyAndHeaders(model);
418 - if (!auth.ok || !auth.apiKey) return { done: false, blocked: false, reason: auth.ok ? "no judge api key" : auth.error };
419 - const response = await complete(
420 - model,
421 - {
422 - systemPrompt: JUDGE_SYSTEM_PROMPT,
423 - messages: [{
424 - role: "user",
425 - content: [{ type: "text", text: JSON.stringify({ objective: state.objective, criteria: state.criteria, latestAssistantResponse: latestAssistantTail(ctx, settings.tailBytes) }, null, 2) }],
579 + if (!auth.ok || !auth.apiKey)
580 + return {
581 + done: false,
582 + blocked: false,
583 + reason: auth.ok ? "no judge api key" : auth.error,
584 + };
585 + const judgeContext = {
586 + systemPrompt: JUDGE_SYSTEM_PROMPT,
587 + messages: [
589 + role: "user" as const,
590 + content: [
592 + type: "text" as const,
593 + text: JSON.stringify(
595 + objective: state.objective,
596 + criteria: state.criteria,
597 + latestAssistantResponse: latestAssistantTail(
598 + ctx,
599 + settings.tailBytes,
600 + ),
601 + },
602 + null,
603 + 2,
604 + ),
605 + },
606 + ],
426 607 timestamp: Date.now(),
427 - }],
428 - },
429 - { apiKey: auth.apiKey, headers: auth.headers, signal: ctx.signal, reasoning: "off", cacheRetention: "none" },
430 - );
431 - const raw = response.content
432 - .filter((part): part is { type: "text"; text: string } => part.type === "text")
433 - .map((part) => part.text)
434 - .join("\n");
435 - return parseJudge(raw);
608 + },
609 + ],
610 + };
611 + let lastError: unknown;
612 + for (let attempt = 0; attempt < JUDGE_ATTEMPTS; attempt++) {
613 + try {
614 + const response = await complete(model, judgeContext, {
615 + apiKey: auth.apiKey,
616 + headers: auth.headers,
617 + signal: ctx.signal,
618 + reasoning: "off",
619 + cacheRetention: "none",
620 + });
621 + const raw = response.content
622 + .filter(
623 + (part): part is { type: "text"; text: string } =>
624 + part.type === "text",
626 + .map((part) => part.text)
627 + .join("\n");
628 + return parseJudge(raw);
629 + } catch (error) {
630 + lastError = error;
633 + throw lastError;
438 636 function isStaleContextError(error: unknown): boolean {
439 - return error instanceof Error && error.message.includes("extension ctx is stale");
637 + return (
638 + error instanceof Error && error.message.includes("extension ctx is stale")
639 + );
442 -function sendWhenIdle(pi: ExtensionAPI, ctx: ExtensionContext, state: GoalState, scheduled: { value: boolean }, attempt = 0) {
443 - setTimeout(() => {
444 - let latest: GoalState | undefined;
445 - try { latest = latestGoalState(ctx); }
446 - catch (error) { if (isStaleContextError(error)) { scheduled.value = false; return; } throw error; }
447 - if (!latest || latest.id !== state.id || latest.status !== "active" || !ownsGoal(ctx, latest)) {
448 - scheduled.value = false;
449 - return;
450 - }
451 - if (!ctx.isIdle() || ctx.hasPendingMessages()) {
452 - if (attempt >= CONTINUATION_MAX_ATTEMPTS) {
642 +function sendWhenIdle(
643 + pi: ExtensionAPI,
644 + ctx: ExtensionContext,
645 + state: GoalState,
646 + scheduled: { value: boolean },
647 + attempt = 0,
648 +) {
649 + setTimeout(
650 + () => {
651 + let latest: GoalState | undefined;
652 + try {
653 + latest = latestGoalState(ctx);
654 + } catch (error) {
655 + if (isStaleContextError(error)) {
656 + scheduled.value = false;
657 + return;
659 + throw error;
661 + if (
662 + !latest ||
663 + latest.id !== state.id ||
664 + latest.status !== "active" ||
665 + !ownsGoal(ctx, latest)
666 + ) {
453 667 scheduled.value = false;
454 668 return;
456 - return sendWhenIdle(pi, ctx, latest, scheduled, attempt + 1);
457 - }
458 - scheduled.value = false;
459 - pi.sendUserMessage(continuationPrompt(latest));
460 - }, attempt === 0 ? 0 : CONTINUATION_RETRY_MS);
670 + if (!ctx.isIdle() || ctx.hasPendingMessages()) {
671 + if (attempt >= CONTINUATION_MAX_ATTEMPTS) {
672 + scheduled.value = false;
673 + return;
675 + return sendWhenIdle(pi, ctx, latest, scheduled, attempt + 1);
677 + scheduled.value = false;
678 + pi.sendUserMessage(continuationPrompt(latest));
679 + },
680 + attempt === 0 ? 0 : CONTINUATION_RETRY_MS,
681 + );
463 -function appendProbe(pi: ExtensionAPI, event: string, data: Record<string, unknown> = {}) {
684 +function appendProbe(
685 + pi: ExtensionAPI,
686 + event: string,
687 + data: Record<string, unknown> = {},
688 +) {
464 689 pi.appendEntry(PROBE_ENTRY, { event, ...data, timestamp: Date.now() });
467 -function sendProbeContinuation(pi: ExtensionAPI, ctx: ExtensionContext, state: ProbeState) {
692 +function sendProbeContinuation(
693 + pi: ExtensionAPI,
694 + ctx: ExtensionContext,
695 + state: ProbeState,
696 +) {
468 697 if (!state.active || state.dispatched) return;
469 698 if (!ctx.isIdle()) {
470 699 appendProbe(pi, "dispatch-skipped-busy", { mode: state.mode });
479 708 pi.sendUserMessage(PROBE_MESSAGE);
482 -function scheduleProbeContinuation(pi: ExtensionAPI, ctx: ExtensionContext, state: ProbeState) {
711 +function scheduleProbeContinuation(
712 + pi: ExtensionAPI,
713 + ctx: ExtensionContext,
714 + state: ProbeState,
715 +) {
483 716 if (!state.active || state.scheduled) return;
484 717 state.scheduled = true;
485 718 appendProbe(pi, "persisted-before-schedule", { mode: state.mode });
486 719 if (state.mode === "direct") return sendProbeContinuation(pi, ctx, state);
487 - if (state.mode === "microtask") return queueMicrotask(() => sendProbeContinuation(pi, ctx, state));
720 + if (state.mode === "microtask")
721 + return queueMicrotask(() => sendProbeContinuation(pi, ctx, state));
488 722 setTimeout(() => sendProbeContinuation(pi, ctx, state), 0);
491 725 function parseProbeMode(args: string): ProbeMode | undefined {
492 726 const mode = args.trim();
493 - if (mode === "direct" || mode === "microtask" || mode === "timeout") return mode;
727 + if (mode === "direct" || mode === "microtask" || mode === "timeout")
728 + return mode;
494 729 return undefined;
497 732 export default function goalExtension(pi: ExtensionAPI) {
733 + pi.registerMessageRenderer(GOAL_JUDGE_ENTRY, (message, _options, theme) => {
734 + const data = isRecord(message.details) ? message.details : {};
735 + const verdict = isRecord(data.verdict)
736 + ? (data.verdict as JudgeVerdict)
737 + : undefined;
738 + const color: ThemeColor = verdict?.done
739 + ? "success"
740 + : verdict?.blocked
741 + ? "warning"
742 + : "accent";
743 + return new Text(theme.fg(color, `◇ ${message.content}`), 0, 0);
744 + });
498 746 let state: GoalState | undefined;
499 - let continuationScheduled = { value: false };
747 + const continuationScheduled = { value: false };
500 748 let probe: ProbeState | undefined;
501 - const syncUpdateGoalTool = () => setToolActive(pi, "update_goal", canUseUpdateGoal(state));
749 + const syncUpdateGoalTool = () =>
750 + setToolActive(pi, "update_goal", canUseUpdateGoal(state));
503 752 pi.on("session_start", (_event, ctx) => {
504 753 state = latestGoalState(ctx);
555 804 updateUI(pi, ctx, state);
556 805 syncUpdateGoalTool();
557 - if (!state || state.status !== "active" || !ownsGoal(ctx, state) || continuationScheduled.value) return;
806 + if (
807 + !state ||
808 + state.status !== "active" ||
809 + !ownsGoal(ctx, state) ||
810 + continuationScheduled.value
812 + return;
558 813 const judged = state;
559 814 appendJudge(pi, "start", judged);
560 815 updateUI(pi, ctx, judged, "judging");
563 818 verdict = await judgeGoal(ctx, judged, loadSettings(pi));
564 819 } catch (err) {
565 - verdict = { done: false, blocked: false, reason: `judge failed: ${(err as Error).message}` };
820 + verdict = {
821 + done: false,
822 + blocked: false,
823 + reason: `judge failed: ${(err as Error).message}`,
824 + };
567 826 if (state !== judged || state.status !== "active") return;
568 827 appendJudge(pi, "verdict", judged, verdict);
569 - if (ctx.hasUI && verdict.reason.startsWith("judge failed:")) ctx.ui.notify(`Goal judge verdict: continue · ${verdict.reason}`, "warning");
828 + if (ctx.hasUI && verdict.reason.startsWith("judge failed:"))
829 + ctx.ui.notify(
830 + `Goal judge verdict: continue · ${verdict.reason}`,
831 + "warning",
832 + );
570 833 if (verdict.done) {
571 - state = updateGoal(pi, judged, { status: "complete", lastReason: verdict.reason });
834 + state = updateGoal(pi, judged, {
835 + status: "complete",
836 + lastReason: verdict.reason,
837 + });
572 838 updateUI(pi, ctx, state);
573 839 syncUpdateGoalTool();
574 - if (ctx.hasUI) ctx.ui.notify(`✓ Goal complete: ${verdict.reason}`, "info");
840 + if (ctx.hasUI)
841 + ctx.ui.notify(`✓ Goal complete: ${verdict.reason}`, "info");
575 842 return;
577 844 if (verdict.blocked) {
578 - state = updateGoal(pi, judged, { status: "blocked", lastReason: verdict.reason });
845 + state = updateGoal(pi, judged, {
846 + status: "blocked",
847 + lastReason: verdict.reason,
848 + });
579 849 updateUI(pi, ctx, state);
580 850 syncUpdateGoalTool();
581 - if (ctx.hasUI) ctx.ui.notify(`Goal blocked: ${verdict.reason}`, "warning");
851 + if (ctx.hasUI)
852 + ctx.ui.notify(`Goal blocked: ${verdict.reason}`, "warning");
582 853 return;
584 855 const turnsUsed = judged.turnsUsed + 1;
585 856 if (turnsUsed >= judged.maxTurns) {
586 - state = updateGoal(pi, judged, { status: "paused", turnsUsed, lastReason: `turn budget exhausted: ${verdict.reason}` });
857 + state = updateGoal(pi, judged, {
858 + status: "paused",
859 + turnsUsed,
860 + lastReason: `turn budget exhausted: ${verdict.reason}`,
861 + });
587 862 updateUI(pi, ctx, state);
588 863 syncUpdateGoalTool();
589 - if (ctx.hasUI) ctx.ui.notify(`Goal paused — ${turnsUsed}/${judged.maxTurns} turns used`, "warning");
864 + if (ctx.hasUI)
865 + ctx.ui.notify(
866 + `Goal paused — ${turnsUsed}/${judged.maxTurns} turns used`,
867 + "warning",
868 + );
590 869 return;
592 871 state = updateGoal(pi, judged, { turnsUsed, lastReason: verdict.reason });
600 879 name: "update_goal",
601 880 label: "Update Goal",
602 881 description: "Mark the active /goal complete or blocked.",
603 - promptSnippet: "Mark the active /goal complete or blocked after judge review.",
882 + promptSnippet:
883 + "Mark the active /goal complete or blocked after judge review.",
604 884 promptGuidelines: [
605 885 "Use update_goal only after a judge review has run and the active /goal is actually complete or genuinely blocked.",
606 886 "Before update_goal complete, verify the objective and all criteria against current files, tests, commands, or deliverables.",
610 890 reason: Type.String(),
612 892 async execute(_toolCallId, params, _signal, _onUpdate, ctx) {
613 - if (!state || state.status !== "active") throw new Error("No active goal.");
614 - if (!ownsGoal(ctx, state)) throw new Error("Goal belongs to another session.");
615 - if (!canUseUpdateGoal(state)) throw new Error("Goal has not been judged yet.");
616 - state = updateGoal(pi, state, { status: params.status, lastReason: params.reason });
893 + if (!state || state.status !== "active")
894 + throw new Error("No active goal.");
895 + if (!ownsGoal(ctx, state))
896 + throw new Error("Goal belongs to another session.");
897 + if (!canUseUpdateGoal(state))
898 + throw new Error("Goal has not been judged yet.");
899 + state = updateGoal(pi, state, {
900 + status: params.status,
901 + lastReason: params.reason,
902 + });
617 903 updateUI(pi, ctx, state);
618 904 syncUpdateGoalTool();
619 - return { content: [{ type: "text", text: `Goal marked ${params.status}: ${params.reason}` }], details: { goal: state } };
905 + return {
906 + content: [
908 + type: "text",
909 + text: `Goal marked ${params.status}: ${params.reason}`,
910 + },
911 + ],
912 + details: { goal: state },
913 + };
623 917 pi.registerCommand("goal", {
624 918 description: "Persistent goal loop. Usage: /goal <objective>",
625 - getArgumentCompletions: (argumentPrefix) => goalArgumentCompletions(argumentPrefix, state),
919 + getArgumentCompletions: (argumentPrefix) =>
920 + goalArgumentCompletions(argumentPrefix, state),
626 921 handler: (args, ctx) => {
627 922 const input = args.trim();
628 923 const firstSpace = input.search(/\s/);
629 924 const cmd = firstSpace === -1 ? input : input.slice(0, firstSpace);
630 925 const rest = firstSpace === -1 ? "" : input.slice(firstSpace).trim();
631 926 if (!input || cmd === "status") {
632 - ctx.ui.notify(state ? statusLine(state, currentSessionId(ctx)) : "No active goal.", state ? "info" : "warning");
927 + ctx.ui.notify(
928 + state ? statusLine(state, currentSessionId(ctx)) : "No active goal.",
929 + state ? "info" : "warning",
930 + );
633 931 updateUI(pi, ctx, state);
634 932 return;
636 934 if (cmd === "pause") {
637 935 if (!ownedGoalOrNotify(ctx, state)) return;
638 - if (state.status !== "active") return ctx.ui.notify("Goal is not active.", "warning");
936 + if (state.status !== "active")
937 + return ctx.ui.notify("Goal is not active.", "warning");
639 938 state = updateGoal(pi, state, { status: "paused" });
640 939 updateUI(pi, ctx, state);
641 940 syncUpdateGoalTool();
645 944 if (cmd === "resume") {
646 945 if (!state) return ctx.ui.notify("No goal to resume.", "warning");
647 - const rebinding = state.sessionId && state.sessionId !== currentSessionId(ctx);
648 - state = updateGoal(pi, state, { status: "active", turnsUsed: 0, sessionId: currentSessionId(ctx) });
946 + const rebinding =
947 + state.sessionId && state.sessionId !== currentSessionId(ctx);
948 + state = updateGoal(pi, state, {
949 + status: "active",
950 + turnsUsed: 0,
951 + sessionId: currentSessionId(ctx),
952 + });
649 953 updateUI(pi, ctx, state);
650 954 syncUpdateGoalTool();
651 - ctx.ui.notify(rebinding ? "Goal resumed in this session" : "Goal resumed", rebinding ? "warning" : "info");
955 + ctx.ui.notify(
956 + rebinding ? "Goal resumed in this session" : "Goal resumed",
957 + rebinding ? "warning" : "info",
958 + );
652 959 continuationScheduled.value = true;
653 960 sendWhenIdle(pi, ctx, state, continuationScheduled);
654 961 return;
656 963 if (cmd === "complete") {
657 964 if (!ownedGoalOrNotify(ctx, state)) return;
658 965 const reason = rest.trim() || "marked complete by user";
659 - state = updateGoal(pi, state, { status: "complete", lastReason: reason });
966 + state = updateGoal(pi, state, {
967 + status: "complete",
968 + lastReason: reason,
969 + });
660 970 updateUI(pi, ctx, state);
661 971 syncUpdateGoalTool();
662 972 ctx.ui.notify("Goal complete", "info");
665 975 if (cmd === "clear") {
666 976 if (!ownedGoalOrNotify(ctx, state)) return;
667 - state = updateGoal(pi, state, { status: "paused", lastReason: "cleared" });
977 + state = updateGoal(pi, state, {
978 + status: "paused",
979 + lastReason: "cleared",
980 + });
668 981 state = undefined;
669 - pi.appendEntry(GOAL_STATE_ENTRY, { cleared: true, updatedAt: Date.now() });
982 + pi.appendEntry(GOAL_STATE_ENTRY, {
983 + cleared: true,
984 + updatedAt: Date.now(),
985 + });
670 986 updateUI(pi, ctx, state);
671 987 syncUpdateGoalTool();
672 988 ctx.ui.notify("Goal cleared", "info");
675 991 if (cmd === "add") {
676 992 if (!ownedGoalOrNotify(ctx, state)) return;
677 - if (!rest.trim()) return ctx.ui.notify("Usage: /goal add <criterion>", "error");
678 - state = updateGoal(pi, state, { criteria: [...state.criteria, rest.trim()] });
993 + if (!rest.trim())
994 + return ctx.ui.notify("Usage: /goal add <criterion>", "error");
995 + state = updateGoal(pi, state, {
996 + criteria: [...state.criteria, rest.trim()],
997 + });
679 998 updateUI(pi, ctx, state);
680 999 syncUpdateGoalTool();
681 1000 ctx.ui.notify("Goal criterion added", "info");
684 1003 if (cmd === "remove") {
685 1004 if (!ownedGoalOrNotify(ctx, state)) return;
686 1005 const index = Number(rest.trim()) - 1;
687 - if (!Number.isInteger(index) || index < 0 || index >= state.criteria.length) return ctx.ui.notify("Usage: /goal remove <n>", "error");
688 - state = updateGoal(pi, state, { criteria: state.criteria.filter((_, i) => i !== index) });
1006 + if (
1007 + !Number.isInteger(index) ||
1008 + index < 0 ||
1009 + index >= state.criteria.length
1011 + return ctx.ui.notify("Usage: /goal remove <n>", "error");
1012 + state = updateGoal(pi, state, {
1013 + criteria: state.criteria.filter((_, i) => i !== index),
1014 + });
689 1015 updateUI(pi, ctx, state);
690 1016 syncUpdateGoalTool();
691 1017 ctx.ui.notify("Goal criterion removed", "info");
692 1018 return;
694 1020 if (cmd === "criteria") {
695 - ctx.ui.notify(state ? criteriaText(state) : "No active goal.", state ? "info" : "warning");
1021 + ctx.ui.notify(
1022 + state ? criteriaText(state) : "No active goal.",
1023 + state ? "info" : "warning",
1024 + );
696 1025 return;
698 1027 if (cmd === "clear-criteria") {
703 1032 ctx.ui.notify("Goal criteria cleared", "info");
704 1033 return;
706 - if (!ctx.isIdle()) return ctx.ui.notify("Agent is running — use /goal status, pause, resume, complete, clear, add, remove, criteria, or clear-criteria.", "warning");
1035 + if (!ctx.isIdle())
1036 + return ctx.ui.notify(
1037 + "Agent is running — use /goal status, pause, resume, complete, clear, add, remove, criteria, or clear-criteria.",
1038 + "warning",
1039 + );
707 1040 const settings = loadSettings(pi);
708 1041 state = {
709 1042 id: crypto.randomUUID(),
729 1062 description: "Internal /goal continuation scheduling probe",
730 1063 handler: (args, ctx) => {
731 1064 const mode = parseProbeMode(args);
732 - if (!mode) return ctx.ui.notify("Usage: /goal-probe direct|microtask|timeout", "error");
1065 + if (!mode)
1066 + return ctx.ui.notify(
1067 + "Usage: /goal-probe direct|microtask|timeout",
1068 + "error",
1069 + );
733 1070 probe = { mode, active: true, scheduled: false, dispatched: false };
734 1071 appendProbe(pi, "start", { mode });
735 1072 ctx.ui.notify(`goal probe armed: ${mode}`, "info");
750 1087 latestGoalState,
751 1088 loadSettings,
752 1089 judgeRunningText,
1090 + judgeDisplayReason,
753 1091 parseJudge,
754 1092 parseModelRef,
755 1093 extractStandingGoalObjective,

extensions/goal/package.json

Modified
2 - "name": "@bugabinga/pi-ext-goal",
3 - "private": true,
4 - "type": "module",
5 - "main": "index.ts",
6 - "peerDependencies": {
7 - "@earendil-works/pi-ai": "*",
8 - "@earendil-works/pi-coding-agent": "*",
9 - "@earendil-works/pi-tui": "*",
10 - "typebox": "*"
11 - },
12 - "devDependencies": {
13 - "@marcfargas/pi-test-harness": "^0.6.1"
14 - },
15 - "scripts": {
16 - "bench": "bun bench/goal-state.bench.ts",
17 - "test:harness": "bun test ./__tests__/harness.test.ts",
18 - "test": "bun test ./__tests__/*.test.ts"
19 - },
20 - "license": "MIT",
21 - "description": "Persistent /goal loop for Pi.",
22 - "keywords": ["pi", "pi-extension", "goal"]
2 + "name": "@bugabinga/pi-ext-goal",
3 + "private": true,
4 + "type": "module",
5 + "main": "index.ts",
6 + "peerDependencies": {
7 + "@earendil-works/pi-ai": "*",
8 + "@earendil-works/pi-coding-agent": "*",
9 + "@earendil-works/pi-tui": "*",
10 + "typebox": "*"
11 + },
12 + "devDependencies": {
13 + "@marcfargas/pi-test-harness": "^0.6.1"
14 + },
15 + "scripts": {
16 + "bench": "bun bench/goal-state.bench.ts",
17 + "test:harness": "bun test ./__tests__/harness.test.ts",
18 + "test": "bun test ./__tests__/*.test.ts"
19 + },
20 + "license": "MIT",
21 + "description": "Persistent /goal loop for Pi.",
22 + "keywords": [
23 + "pi",
24 + "pi-extension",
25 + "goal"
26 + ]