repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by admin

feat(witness): run manual text observers

90c6c0bb0023a22fc42b31c4cf29fa509eb663ce

Oliver Krylow <o.krylow@isp-insoft.de> ·

SSH signature · not checked

Browse filesCompare with HEAD

Changed files (10)

extensions/witness/PLAN.md

Modified
30 30 ## Slice 2: Run one manual text observer
32 -- [ ] Add `/witness run <id>` for a manually triggered text observer.
33 -- [ ] Resolve the configured model, authentication, thinking level, prompt, and bounds without silent fallback.
34 -- [ ] Create a minimal parent-linked child session in the parent's session store.
35 -- [ ] Prevent child resource inheritance and Witness recursion.
36 -- [ ] Persist real child assistant usage without copying usage into the parent.
37 -- [ ] Deliver sanitized text through the shared Witness widget.
38 -- [ ] Abort and invalidate children on timeout, disablement, reload, tree navigation, session replacement, and shutdown.
39 -- [ ] Test persistent and ephemeral parents, unavailable models, extension-registered providers, stale completions, and child-session attribution.
32 +- [x] Add `/witness run <id>` for a manually triggered text observer.
33 +- [x] Resolve the configured model, authentication, thinking level, prompt, and bounds without silent fallback.
34 +- [x] Create a minimal parent-linked child session in the parent's session store.
35 +- [x] Prevent child resource inheritance and Witness recursion.
36 +- [x] Persist real child assistant usage without copying usage into the parent.
37 +- [x] Deliver sanitized text through the shared Witness widget.
38 +- [x] Abort and invalidate children on timeout, disablement, reload, tree navigation, session replacement, and shutdown.
39 +- [x] Test persistent and ephemeral parents, unavailable models, extension-registered providers, stale completions, and child-session attribution.
40 40 - [ ] Compare warm and cold latency against direct `ctx.modelRegistry.complete()` on representative fast models.
41 -- [ ] Run `mise run //extensions/witness:test` and `mise run //extensions/witness:check`.
41 +- [x] Run `mise run //extensions/witness:test` and `mise run //extensions/witness:check`.
43 43 Exit condition: one configured observer completes a real manual child run with visible output and accounted usage.
143 143 Exit condition: Witness hosts migrated policies independently, accounts for their work, and remains quiet when no definitions are installed.
145 -## Deferred
145 +## Optional future policy
147 -- Verification-audit policy remains deferred until migrated policies establish trustworthy evidence and attention mechanisms.
148 -- Arbitrary scripting, observer dependencies, observer-to-observer conversations, inherited parent resources, and custom renderer code remain out of scope.
147 +- Verification-audit remains an uncommitted candidate until migrated policies establish trustworthy evidence and attention mechanisms.
149 +## Non-goals
151 +- Arbitrary scripting, observer dependencies, observer-to-observer conversations, inherited parent resources, and custom renderer code are out of scope.
149 152 - Publishing a remote JSON Schema remains optional because runtime validation and the creation UI are authoritative.

extensions/witness/README.md

Modified
10 10 ## Current state
12 -Witness discovers and validates observer definitions without running models.
12 +Witness discovers, validates, and manually runs text observer definitions.
13 13 It ships no observers or templates by default.
15 15 Active user definitions live in `~/.pi/agent/witness/observers/`.
16 16 Trusted project definitions live in `.pi/witness/` and completely replace matching user IDs.
18 18 Use `/witness list`, `/witness show <id>`, and `/witness validate` to inspect the resolved configuration.
19 +Use `/witness run <id>` to execute one ready observer in an isolated, tool-free child session and deliver its text through the shared widget.
20 -The first slice accepts manual text observers with empty context and widget delivery:
21 +The current execution slice accepts manual text observers with empty context and widget delivery:
22 23 ```markdown
45 46 Witness applies defaults of 30 seconds, 1,024 output tokens, zero retries, and zero tool calls under `agent.limits`.
46 47 Unsupported future trigger, context, capability, result, and delivery forms remain disabled until their implementation slice lands.
47 -A `ready` definition has a catalogued model and valid slice-1 structure; authentication and execution readiness begin in slice 2.
48 +A `ready` definition has a catalogued model and valid current structure.
49 +Execution rejects unavailable authentication or unsupported thinking levels without silently substituting either.
49 51 ## Performance

extensions/witness/SPEC.md

Modified
3 3 # Witness specification
5 5 ## Status
7 7 This document defines the intended Witness behavior.
8 -Implementation slice 1 discovers, validates, and explains definitions without running models.
8 +Implementation slice 2 also runs manually triggered text observers with empty context and capabilities in isolated child sessions.
10 10 ## Purpose

extensions/witness/__tests__/execution.test.ts

Added
new file mode 100644
1 +import {
2 + mkdirSync,
3 + mkdtempSync,
4 + readdirSync,
5 + readFileSync,
6 + rmSync,
7 +} from "node:fs";
8 +import { tmpdir } from "node:os";
9 +import path from "node:path";
10 +import {
11 + createAssistantMessageEventStream,
12 + fauxAssistantMessage,
13 + fauxProvider,
14 + fauxText,
15 + fauxToolCall,
16 + InMemoryCredentialStore,
17 + type Provider,
18 + type SimpleStreamOptions,
19 +} from "@earendil-works/pi-ai";
20 +import {
21 + ModelRegistry,
22 + ModelRuntime,
23 + SessionManager,
24 +} from "@earendil-works/pi-coding-agent";
25 +import { afterEach, describe, expect, it, vi } from "vitest";
26 +import { parseObserverDefinition } from "../definitions";
27 +import { runObserver } from "../execution";
29 +const TEST_COST = {
30 + input: 0.1,
31 + output: 0.2,
32 + cacheRead: 0,
33 + cacheWrite: 0,
34 + total: 0.3,
35 +};
37 +function observer(model: string, limits = ""): string {
38 + return `---
39 +id: review
40 +trigger:
41 + manual: true
42 +context:
43 + sources: []
44 +agent:
45 + model: ${model}
46 + thinking: low
47 + tools: []
48 + skills: []
49 + prompts: []
50 + extensions: []
51 +${limits}result:
52 + type: text
53 +delivery:
54 + type: widget
55 +---
57 +Judge only the supplied evidence.`;
60 +async function fixture(persistent: boolean) {
61 + const root = mkdtempSync(path.join(tmpdir(), "witness-execution-"));
62 + const cwd = path.join(root, "project");
63 + const sessionDir = path.join(root, "sessions");
64 + mkdirSync(cwd, { recursive: true });
65 + const faux = fauxProvider({
66 + provider: "witness-test",
67 + models: [
68 + {
69 + id: "observer",
70 + reasoning: true,
71 + maxTokens: 8_192,
72 + cost: { input: 1, output: 2, cacheRead: 0.5, cacheWrite: 0.5 },
73 + },
74 + ],
75 + });
76 + const runtime = await ModelRuntime.create({
77 + credentials: new InMemoryCredentialStore(),
78 + modelsPath: null,
79 + refreshOnCreate: false,
80 + });
81 + runtime.registerNativeProvider(faux.provider);
82 + const registry = new ModelRegistry(runtime);
83 + const parent = persistent
84 + ? SessionManager.create(cwd, sessionDir)
85 + : SessionManager.inMemory(cwd);
86 + if (persistent) {
87 + parent.appendMessage({
88 + role: "user",
89 + content: "Private parent context",
90 + timestamp: Date.now(),
91 + });
92 + parent.appendMessage({
93 + ...response("Completed parent answer"),
94 + provider: "parent-provider",
95 + model: "parent-model",
96 + });
97 + }
98 + const ctx = {
99 + cwd,
100 + modelRegistry: registry,
101 + sessionManager: parent,
102 + } as never;
103 + return { root, cwd, sessionDir, faux, parent, ctx };
106 +function response(text: string) {
107 + return {
108 + ...fauxAssistantMessage(text),
109 + usage: {
110 + input: 13,
111 + output: 7,
112 + cacheRead: 0,
113 + cacheWrite: 0,
114 + totalTokens: 20,
115 + cost: TEST_COST,
116 + },
117 + };
120 +function exactUsageProvider(provider: Provider): Provider {
121 + const message = {
122 + ...response("Persistent result"),
123 + provider: "witness-test",
124 + model: "observer",
125 + };
126 + const stream = () => {
127 + const events = createAssistantMessageEventStream();
128 + queueMicrotask(() => {
129 + events.push({ type: "start", partial: message });
130 + events.push({ type: "done", reason: "stop", message });
131 + });
132 + return events;
133 + };
134 + return { ...provider, stream, streamSimple: stream } as Provider;
137 +function abortUsageProvider(provider: Provider, onStart: () => void): Provider {
138 + const stream = (
139 + _model: unknown,
140 + _context: unknown,
141 + options: SimpleStreamOptions | undefined,
142 + ) => {
143 + const events = createAssistantMessageEventStream();
144 + const partial = {
145 + ...response(""),
146 + provider: "witness-test",
147 + model: "observer",
148 + };
149 + events.push({ type: "start", partial });
150 + onStart();
151 + options?.signal?.addEventListener(
152 + "abort",
153 + () => {
154 + setTimeout(() => {
155 + const message = {
156 + ...response("Cancelled child"),
157 + provider: "witness-test",
158 + model: "observer",
159 + stopReason: "aborted" as const,
160 + };
161 + events.push({ type: "error", reason: "aborted", error: message });
162 + }, 5);
163 + },
164 + { once: true },
165 + );
166 + return events;
167 + };
168 + return { ...provider, stream, streamSimple: stream } as Provider;
171 +describe("Witness observer execution", () => {
172 + const roots: string[] = [];
173 + afterEach(() => {
174 + for (const root of roots.splice(0))
175 + rmSync(root, { recursive: true, force: true });
176 + });
178 + it("runs with exact policy, no inherited resources, and bounded provider options", async () => {
179 + const f = await fixture(false);
180 + roots.push(f.root);
181 + let request:
182 + | {
183 + systemPrompt?: string;
184 + tools: readonly unknown[];
185 + options?: SimpleStreamOptions;
187 + | undefined;
188 + f.faux.setResponses([
189 + (context, options) => {
190 + request = {
191 + systemPrompt: context.systemPrompt,
192 + tools: context.tools ?? [],
193 + options,
194 + };
195 + return response("Observed result");
196 + },
197 + ]);
198 + const definition = parseObserverDefinition(
199 + observer(
200 + "witness-test/observer",
201 + " limits:\n timeoutMs: 5000\n outputTokens: 321\n retries: 0\n toolCalls: 0\n",
202 + ),
203 + );
205 + const result = await runObserver(
206 + definition,
207 + f.ctx,
208 + new AbortController().signal,
209 + );
211 + expect(result.text).toBe("Observed result");
212 + expect(result.message).toMatchObject({
213 + provider: "witness-test",
214 + model: "observer",
215 + usage: { input: expect.any(Number), output: expect.any(Number) },
216 + });
217 + expect(result.message.usage.totalTokens).toBeGreaterThan(0);
218 + expect(result.sessionFile).toBeUndefined();
219 + expect(request).toMatchObject({
220 + systemPrompt: `Judge only the supplied evidence.\nCurrent working directory: ${f.cwd.replace(/\\/g, "/")}\n`,
221 + tools: [],
222 + options: { maxTokens: 321, maxRetries: 0 },
223 + });
224 + });
226 + it("persists a parent-linked child with real assistant usage only in the child", async () => {
227 + const f = await fixture(true);
228 + roots.push(f.root);
229 + f.ctx.modelRegistry.registerProvider(exactUsageProvider(f.faux.provider));
230 + const parentFile = f.parent.getSessionFile();
231 + const parentBefore = readFileSync(parentFile as string, "utf8");
232 + const result = await runObserver(
233 + parseObserverDefinition(observer("witness-test/observer")),
234 + f.ctx,
235 + new AbortController().signal,
236 + );
238 + expect(result.message.usage).toEqual({
239 + input: 13,
240 + output: 7,
241 + cacheRead: 0,
242 + cacheWrite: 0,
243 + totalTokens: 20,
244 + cost: TEST_COST,
245 + });
246 + expect(result.sessionFile).toBeDefined();
247 + expect(result.sessionFile).not.toBe(parentFile);
248 + const entries = readFileSync(result.sessionFile as string, "utf8")
249 + .trim()
250 + .split("\n")
251 + .map((line) => JSON.parse(line));
252 + expect(entries[0]).toMatchObject({
253 + type: "session",
254 + parentSession: parentFile,
255 + });
256 + expect(entries).toContainEqual(
257 + expect.objectContaining({
258 + type: "thinking_level_change",
259 + thinkingLevel: "low",
260 + }),
261 + );
262 + expect(entries).toContainEqual(
263 + expect.objectContaining({
264 + type: "custom",
265 + customType: "witness",
266 + data: expect.objectContaining({ observerId: "review" }),
267 + }),
268 + );
269 + expect(entries).toContainEqual(
270 + expect.objectContaining({
271 + type: "message",
272 + message: expect.objectContaining({
273 + role: "assistant",
274 + provider: result.message.provider,
275 + model: result.message.model,
276 + usage: result.message.usage,
277 + }),
278 + }),
279 + );
280 + const parentAfter = readFileSync(parentFile as string, "utf8");
281 + expect(parentAfter).toBe(parentBefore);
282 + });
284 + it("retains child persistence until cooperative cancellation settles", async () => {
285 + const f = await fixture(true);
286 + roots.push(f.root);
287 + let started: (() => void) | undefined;
288 + const providerStarted = new Promise<void>((resolve) => {
289 + started = resolve;
290 + });
291 + f.ctx.modelRegistry.registerProvider(
292 + abortUsageProvider(f.faux.provider, () => started?.()),
293 + );
294 + const controller = new AbortController();
295 + const running = runObserver(
296 + parseObserverDefinition(observer("witness-test/observer")),
297 + f.ctx,
298 + controller.signal,
299 + );
300 + await providerStarted;
302 + controller.abort(new Error("cancelled by test"));
303 + await expect(running).rejects.toThrow("cancelled by test");
304 + await expect
305 + .poll(() =>
306 + readdirSync(f.sessionDir).filter((file) => file.endsWith(".jsonl")),
308 + .toHaveLength(2);
309 + const childFile = readdirSync(f.sessionDir)
310 + .map((file) => path.join(f.sessionDir, file))
311 + .find((file) => file !== f.parent.getSessionFile());
312 + const entries = readFileSync(childFile as string, "utf8")
313 + .trim()
314 + .split("\n")
315 + .map((line) => JSON.parse(line));
316 + expect(entries).toContainEqual(
317 + expect.objectContaining({
318 + type: "message",
319 + message: expect.objectContaining({
320 + role: "assistant",
321 + stopReason: "aborted",
322 + usage: expect.objectContaining({ cost: TEST_COST }),
323 + }),
324 + }),
325 + );
326 + });
328 + it("bridges resolved authentication and provider environment exactly", async () => {
329 + const f = await fixture(false);
330 + roots.push(f.root);
331 + let options: Record<string, unknown> | undefined;
332 + f.faux.setResponses([
333 + (_context, requestOptions) => {
334 + options = requestOptions as Record<string, unknown>;
335 + return response("Authenticated result");
336 + },
337 + ]);
338 + vi.spyOn(f.ctx.modelRegistry, "getApiKeyAndHeaders").mockResolvedValue({
339 + ok: true,
340 + apiKey: "fake",
341 + headers: { "X-Test-Auth": "header-value" },
342 + baseUrl: "https://provider.invalid/v1",
343 + env: { TEST_REGION: "test-region" },
344 + });
346 + await runObserver(
347 + parseObserverDefinition(observer("witness-test/observer")),
348 + f.ctx,
349 + new AbortController().signal,
350 + );
352 + expect(options).toMatchObject({
353 + apiKey: "fake",
354 + headers: { "X-Test-Auth": "header-value" },
355 + baseUrl: "https://provider.invalid/v1",
356 + env: { TEST_REGION: "test-region" },
357 + });
358 + });
360 + it("rejects unsupported thinking without making a provider request", async () => {
361 + const f = await fixture(false);
362 + roots.push(f.root);
363 + const model = { ...f.faux.getModel(), reasoning: false };
364 + vi.spyOn(f.ctx.modelRegistry, "find").mockReturnValue(model);
366 + await expect(
367 + runObserver(
368 + parseObserverDefinition(observer("witness-test/observer")),
369 + f.ctx,
370 + new AbortController().signal,
371 + ),
372 + ).rejects.toThrow("Thinking level");
373 + expect(f.faux.state.callCount).toBe(0);
374 + });
376 + it("stops after and rejects a first-turn tool call", async () => {
377 + const f = await fixture(false);
378 + roots.push(f.root);
379 + f.faux.setResponses([
380 + fauxAssistantMessage(
381 + fauxToolCall("unavailable", {}, { id: "forbidden-call" }),
382 + { stopReason: "toolUse" },
383 + ),
384 + fauxAssistantMessage(fauxText("Must not run")),
385 + ]);
387 + await expect(
388 + runObserver(
389 + parseObserverDefinition(observer("witness-test/observer")),
390 + f.ctx,
391 + new AbortController().signal,
392 + ),
393 + ).rejects.toThrow("toolUse");
394 + expect(f.faux.state.callCount).toBe(1);
395 + });
397 + it("does not let provider options exceed the model output ceiling", async () => {
398 + const f = await fixture(false);
399 + roots.push(f.root);
400 + let maxTokens: number | undefined;
401 + f.faux.setResponses([
402 + (_context, options) => {
403 + maxTokens = options?.maxTokens;
404 + return response("Bounded result");
405 + },
406 + ]);
407 + const definition = parseObserverDefinition(
408 + observer("witness-test/observer", " limits:\n outputTokens: 9000\n"),
409 + );
411 + await runObserver(definition, f.ctx, new AbortController().signal);
413 + expect(maxTokens).toBe(8_192);
414 + });
416 + it("bounds an uncooperative provider wait with the execution signal", async () => {
417 + const f = await fixture(false);
418 + roots.push(f.root);
419 + f.faux.setResponses([async () => await new Promise(() => undefined)]);
420 + const controller = new AbortController();
421 + setTimeout(() => controller.abort(new Error("provider timeout")), 10);
423 + await expect(
424 + runObserver(
425 + parseObserverDefinition(observer("witness-test/observer")),
426 + f.ctx,
427 + controller.signal,
428 + ),
429 + ).rejects.toThrow("provider timeout");
430 + expect(f.faux.state.callCount).toBe(1);
431 + });
433 + it("bounds authentication waiting with the execution signal", async () => {
434 + const f = await fixture(false);
435 + roots.push(f.root);
436 + vi.spyOn(f.ctx.modelRegistry, "getApiKeyAndHeaders").mockReturnValue(
437 + new Promise(() => undefined),
438 + );
439 + const controller = new AbortController();
440 + setTimeout(() => controller.abort(new Error("auth timeout")), 10);
442 + await expect(
443 + runObserver(
444 + parseObserverDefinition(observer("witness-test/observer")),
445 + f.ctx,
446 + controller.signal,
447 + ),
448 + ).rejects.toThrow("auth timeout");
449 + expect(f.faux.state.callCount).toBe(0);
450 + });
452 + it("fails before execution when authentication is unavailable", async () => {
453 + const f = await fixture(false);
454 + roots.push(f.root);
455 + vi.spyOn(f.ctx.modelRegistry, "getApiKeyAndHeaders").mockResolvedValue({
456 + ok: false,
457 + error: "missing key",
458 + });
460 + await expect(
461 + runObserver(
462 + parseObserverDefinition(observer("witness-test/observer")),
463 + f.ctx,
464 + new AbortController().signal,
465 + ),
466 + ).rejects.toThrow("Authentication");
467 + expect(f.faux.state.callCount).toBe(0);
468 + });
469 +});

extensions/witness/__tests__/harness.test.ts

Modified
1 1 import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
2 2 import { tmpdir } from "node:os";
3 3 import path from "node:path";
4 +import { fauxAssistantMessage, fauxProvider } from "@earendil-works/pi-ai";
4 5 import { afterEach, describe, expect, it, vi } from "vitest";
5 6 import {
6 7 createTestSession,
166 167 ` ${field}: [example]`,
168 169 expect(() => parseObserverDefinition(source)).toThrow(
169 - "must be empty because child capabilities start in slice 2",
170 + "must be empty in the current execution slice",
172 173 parseObserverDefinition(source);
400 + it("runs an extension-registered model and delivers its text through the widget", async () => {
401 + t = await createTestSession({ extensionFactories: [witnessExtension] });
402 + const definitions = path.join(
403 + t.env.HOME as string,
404 + ".pi",
405 + "agent",
406 + "witness",
407 + "observers",
408 + );
409 + await mkdir(definitions, { recursive: true });
410 + const runner = t.session.extensionRunner;
411 + const ctx = runner.createCommandContext();
412 + const faux = fauxProvider({
413 + provider: "witness-command-test",
414 + models: [{ id: "observer", reasoning: true }],
415 + });
416 + faux.setResponses([fauxAssistantMessage("Widget result")]);
417 + ctx.modelRegistry.registerProvider(faux.provider);
418 + await writeFile(
419 + path.join(definitions, "review.md"),
420 + observer({ model: "witness-command-test/observer" }),
421 + );
422 + const command = runner.extensions[0]?.commands.get("witness");
423 + const setWidget = vi.spyOn(ctx.ui, "setWidget");
425 + await withProcessEnv(t.env, () => command?.handler("validate", ctx));
426 + await withProcessEnv(t.env, () => command?.handler("run review", ctx));
428 + expect(setWidget).toHaveBeenNthCalledWith(1, "witness", [
429 + "Witness review: running",
430 + ]);
431 + expect(setWidget).toHaveBeenLastCalledWith("witness", ["Widget result"]);
432 + expect(faux.state.callCount).toBe(1);
434 + await writeFile(
435 + path.join(definitions, "review.md"),
436 + observer({
437 + model: "witness-command-test/observer",
438 + prompt: "Changed observer prompt.",
439 + }),
440 + );
441 + await withProcessEnv(t.env, () => command?.handler("validate", ctx));
442 + expect(setWidget).toHaveBeenLastCalledWith("witness", undefined);
443 + });
445 + it("suppresses a child completion after session shutdown", async () => {
446 + t = await createTestSession({ extensionFactories: [witnessExtension] });
447 + const definitions = path.join(
448 + t.env.HOME as string,
449 + ".pi",
450 + "agent",
451 + "witness",
452 + "observers",
453 + );
454 + await mkdir(definitions, { recursive: true });
455 + const runner = t.session.extensionRunner;
456 + const ctx = runner.createCommandContext();
457 + const faux = fauxProvider({
458 + provider: "witness-stale-test",
459 + models: [{ id: "observer", reasoning: true }],
460 + });
461 + let release: (() => void) | undefined;
462 + faux.setResponses([
463 + async () => {
464 + await new Promise<void>((resolve) => {
465 + release = resolve;
466 + });
467 + return fauxAssistantMessage("Stale result");
468 + },
469 + ]);
470 + ctx.modelRegistry.registerProvider(faux.provider);
471 + await writeFile(
472 + path.join(definitions, "review.md"),
473 + observer({ model: "witness-stale-test/observer" }),
474 + );
475 + const command = runner.extensions[0]?.commands.get("witness");
476 + const setWidget = vi.spyOn(ctx.ui, "setWidget");
477 + await withProcessEnv(t.env, () => command?.handler("validate", ctx));
478 + const running = withProcessEnv(t.env, () =>
479 + command?.handler("run review", ctx),
480 + );
481 + await expect.poll(() => faux.state.callCount).toBe(1);
483 + await runner.emit({ type: "session_shutdown", reason: "reload" });
484 + release?.();
485 + await running;
487 + expect(setWidget).toHaveBeenLastCalledWith("witness", undefined);
488 + expect(
489 + setWidget.mock.calls.some((call) =>
490 + JSON.stringify(call).includes("Stale result"),
491 + ),
492 + ).toBe(false);
493 + });
495 + it("clears a running status when validation replaces its definition", async () => {
496 + t = await createTestSession({ extensionFactories: [witnessExtension] });
497 + const definitions = path.join(
498 + t.env.HOME as string,
499 + ".pi",
500 + "agent",
501 + "witness",
502 + "observers",
503 + );
504 + await mkdir(definitions, { recursive: true });
505 + const runner = t.session.extensionRunner;
506 + const ctx = runner.createCommandContext();
507 + const faux = fauxProvider({
508 + provider: "witness-replacement-test",
509 + models: [{ id: "observer", reasoning: true }],
510 + });
511 + let release: (() => void) | undefined;
512 + faux.setResponses([
513 + async () => {
514 + await new Promise<void>((resolve) => {
515 + release = resolve;
516 + });
517 + return fauxAssistantMessage("Obsolete result");
518 + },
519 + ]);
520 + ctx.modelRegistry.registerProvider(faux.provider);
521 + const definitionFile = path.join(definitions, "review.md");
522 + await writeFile(
523 + definitionFile,
524 + observer({ model: "witness-replacement-test/observer" }),
525 + );
526 + const command = runner.extensions[0]?.commands.get("witness");
527 + const setWidget = vi.spyOn(ctx.ui, "setWidget");
528 + await withProcessEnv(t.env, () => command?.handler("validate", ctx));
529 + const running = withProcessEnv(t.env, () =>
530 + command?.handler("run review", ctx),
531 + );
532 + await expect.poll(() => faux.state.callCount).toBe(1);
534 + await writeFile(
535 + definitionFile,
536 + observer({
537 + model: "witness-replacement-test/observer",
538 + prompt: "Replacement prompt.",
539 + }),
540 + );
541 + await withProcessEnv(t.env, () => command?.handler("validate", ctx));
542 + release?.();
543 + await running;
545 + expect(setWidget).toHaveBeenLastCalledWith("witness", undefined);
546 + expect(
547 + setWidget.mock.calls.some((call) =>
548 + JSON.stringify(call).includes("Obsolete result"),
549 + ),
550 + ).toBe(false);
551 + });
399 553 it("reports a discovery failure and permits a clean retry", async () => {
400 554 t = await createTestSession({ extensionFactories: [witnessExtension] });
401 555 const definitions = path.join(

extensions/witness/definitions.ts

Modified
146 146 if (!Array.isArray(value)) fail(field, "must be a list of non-empty strings");
147 147 value.forEach((item, index) => requiredString(item, `${field}[${index}]`));
148 148 if (value.length > 0)
149 - fail(field, "must be empty because child capabilities start in slice 2");
149 + fail(field, "must be empty in the current execution slice");
150 150 return [];

extensions/witness/execution.ts

Added
new file mode 100644
1 +import path from "node:path";
2 +import type {
3 + Api,
4 + AssistantMessage,
5 + Model,
6 + Provider,
7 + SimpleStreamOptions,
8 + StreamOptions,
9 +} from "@earendil-works/pi-ai";
10 +import {
11 + getSupportedThinkingLevels,
12 + InMemoryCredentialStore,
13 +} from "@earendil-works/pi-ai";
14 +import {
15 + createAgentSession,
16 + createExtensionRuntime,
17 + type ExtensionContext,
18 + ModelRuntime,
19 + type ResourceLoader,
20 + SessionManager,
21 + SettingsManager,
22 +} from "@earendil-works/pi-coding-agent";
23 +import type { ObserverDefinition } from "./definitions";
25 +export interface ObserverRunResult {
26 + text: string;
27 + message: AssistantMessage;
28 + sessionFile?: string;
31 +export async function runObserver(
32 + definition: ObserverDefinition,
33 + ctx: ExtensionContext,
34 + signal: AbortSignal,
35 +): Promise<ObserverRunResult> {
36 + throwIfAborted(signal);
37 + const model = resolveModel(definition, ctx);
38 + if (!getSupportedThinkingLevels(model).includes(definition.agent.thinking)) {
39 + throw new Error(
40 + `Thinking level ${JSON.stringify(definition.agent.thinking)} is unavailable for ${JSON.stringify(definition.agent.model)}`,
41 + );
42 + }
43 + const provider = ctx.modelRegistry.getProvider(model.provider);
44 + if (!provider) {
45 + throw new Error(
46 + `Provider ${JSON.stringify(model.provider)} is unavailable`,
47 + );
48 + }
49 + const inheritedAuth = await abortable(
50 + ctx.modelRegistry.getApiKeyAndHeaders(model),
51 + signal,
52 + );
53 + throwIfAborted(signal);
54 + if (!inheritedAuth.ok) {
55 + throw new Error(
56 + `Authentication for ${JSON.stringify(definition.agent.model)} is unavailable`,
57 + );
58 + }
60 + const boundedModel = {
61 + ...model,
62 + maxTokens: Math.min(model.maxTokens, definition.agent.limits.outputTokens),
63 + };
64 + const runtime = await ModelRuntime.create({
65 + credentials: new InMemoryCredentialStore(),
66 + modelsPath: null,
67 + refreshOnCreate: false,
68 + });
69 + runtime.registerNativeProvider(
70 + inheritProvider(provider, boundedModel, inheritedAuth, definition, signal),
71 + );
73 + const sessionManager = createChildSession(definition, ctx);
74 + const settingsManager = SettingsManager.inMemory({
75 + compaction: { enabled: false },
76 + retry: {
77 + enabled: false,
78 + provider: { maxRetries: definition.agent.limits.retries },
79 + },
80 + });
81 + const resources = emptyResources(ctx.cwd, definition.prompt);
82 + throwIfAborted(signal);
83 + const { session } = await createAgentSession({
84 + cwd: ctx.cwd,
85 + model: boundedModel,
86 + thinkingLevel: definition.agent.thinking,
87 + tools: [],
88 + modelRuntime: runtime,
89 + resourceLoader: resources,
90 + settingsManager,
91 + sessionManager,
92 + });
93 + if (session.thinkingLevel !== definition.agent.thinking) {
94 + session.dispose();
95 + throw new Error(
96 + `Thinking level ${JSON.stringify(definition.agent.thinking)} could not be selected exactly`,
97 + );
98 + }
99 + session.agent.shouldStopAfterTurn = () => true;
100 + const abort = () => session.abort();
101 + let prompt: Promise<void> | undefined;
102 + let promptSettled = false;
103 + signal.addEventListener("abort", abort, { once: true });
104 + try {
105 + throwIfAborted(signal);
106 + prompt = session.prompt(
107 + "Run the observer now using only the supplied context.",
108 + );
109 + await abortable(prompt, signal);
110 + promptSettled = true;
111 + throwIfAborted(signal);
113 + const message = [...session.messages]
114 + .reverse()
115 + .find(
116 + (candidate): candidate is AssistantMessage =>
117 + candidate.role === "assistant",
118 + );
119 + if (!message) throw new Error("Observer returned no assistant message");
120 + if (message.stopReason !== "stop") {
121 + throw new Error(`Observer stopped with ${message.stopReason}`);
123 + if (message.content.some((part) => part.type === "toolCall")) {
124 + throw new Error("Observer attempted an unavailable tool call");
126 + const text = message.content
127 + .filter((part) => part.type === "text")
128 + .map((part) => part.text)
129 + .join("")
130 + .trim();
131 + if (!text) throw new Error("Observer returned no text");
132 + return { text, message, sessionFile: sessionManager.getSessionFile() };
133 + } finally {
134 + signal.removeEventListener("abort", abort);
135 + if (prompt && !promptSettled) {
136 + void prompt.then(
137 + () => session.dispose(),
138 + () => session.dispose(),
139 + );
140 + } else {
141 + session.dispose();
146 +function resolveModel(
147 + definition: ObserverDefinition,
148 + ctx: ExtensionContext,
149 +): Model<Api> {
150 + const separator = definition.agent.model.indexOf("/");
151 + const provider = definition.agent.model.slice(0, separator);
152 + const modelId = definition.agent.model.slice(separator + 1);
153 + const model = ctx.modelRegistry.find(provider, modelId);
154 + if (!model) {
155 + throw new Error(
156 + `Model ${JSON.stringify(definition.agent.model)} is unavailable`,
157 + );
159 + return model;
162 +function inheritProvider(
163 + provider: Provider,
164 + model: Model<Api>,
165 + auth: Awaited<
166 + ReturnType<ExtensionContext["modelRegistry"]["getApiKeyAndHeaders"]>
167 + > & {
168 + ok: true;
169 + },
170 + definition: ObserverDefinition,
171 + signal: AbortSignal,
172 +): Provider {
173 + const bound = model.maxTokens;
174 + const bounded = <T extends StreamOptions | SimpleStreamOptions | undefined>(
175 + options: T,
176 + ) =>
177 + ({
178 + ...options,
179 + apiKey: auth.apiKey,
180 + headers: auth.headers,
181 + baseUrl: auth.baseUrl,
182 + maxTokens: Math.min(options?.maxTokens ?? bound, bound),
183 + maxRetries: definition.agent.limits.retries,
184 + signal: options?.signal
185 + ? AbortSignal.any([options.signal, signal])
186 + : signal,
187 + }) as T;
188 + return {
189 + id: provider.id,
190 + name: provider.name,
191 + baseUrl: provider.baseUrl,
192 + headers: provider.headers,
193 + auth: {
194 + apiKey: {
195 + name: "Inherited parent authentication",
196 + check: async () => ({ type: "api_key" }),
197 + resolve: async () => ({
198 + auth: {
199 + apiKey: auth.apiKey,
200 + headers: auth.headers,
201 + baseUrl: auth.baseUrl,
202 + },
203 + env: auth.env,
204 + }),
205 + },
206 + },
207 + getModels: () => [model],
208 + stream: (selected, context, options) =>
209 + provider.stream(selected, context, bounded(options)),
210 + streamSimple: (selected, context, options) =>
211 + provider.streamSimple(selected, context, bounded(options)),
212 + };
215 +function createChildSession(
216 + definition: ObserverDefinition,
217 + ctx: ExtensionContext,
218 +): SessionManager {
219 + const parentSession = ctx.sessionManager.getSessionFile();
220 + if (!parentSession) return SessionManager.inMemory(ctx.cwd);
221 + const child = SessionManager.create(ctx.cwd, path.dirname(parentSession), {
222 + parentSession,
223 + });
224 + child.appendSessionInfo(`Witness: ${definition.id}`);
225 + child.appendCustomEntry("witness", {
226 + observerId: definition.id,
227 + trigger: "manual",
228 + model: definition.agent.model,
229 + });
230 + return child;
233 +function throwIfAborted(signal: AbortSignal): void {
234 + if (signal.aborted)
235 + throw signal.reason ?? new Error("Observer execution aborted");
238 +function abortable<T>(promise: Promise<T>, signal: AbortSignal): Promise<T> {
239 + if (signal.aborted) return Promise.reject(signal.reason);
240 + return new Promise<T>((resolve, reject) => {
241 + const abort = () => reject(signal.reason);
242 + signal.addEventListener("abort", abort, { once: true });
243 + void promise.then(
244 + (value) => {
245 + signal.removeEventListener("abort", abort);
246 + resolve(value);
247 + },
248 + (error) => {
249 + signal.removeEventListener("abort", abort);
250 + reject(error);
251 + },
252 + );
253 + });
256 +function emptyResources(_cwd: string, systemPrompt: string): ResourceLoader {
257 + const runtime = createExtensionRuntime();
258 + return {
259 + getExtensions: () => ({ extensions: [], errors: [], runtime }),
260 + getSkills: () => ({ skills: [], diagnostics: [] }),
261 + getPrompts: () => ({ prompts: [], diagnostics: [] }),
262 + getThemes: () => ({ themes: [], diagnostics: [] }),
263 + getAgentsFiles: () => ({ agentsFiles: [] }),
264 + getSystemPrompt: () => systemPrompt,
265 + getSystemPromptSource: () => undefined,
266 + getAppendSystemPrompt: () => [],
267 + getAppendSystemPromptSources: () => [],
268 + extendResources: () => undefined,
269 + reload: async () => undefined,
270 + };

extensions/witness/index.ts

Modified
10 10 import {
11 11 type DiscoveryResult,
12 12 discoverObservers,
13 + type ObserverDefinition,
13 14 type ResolvedObserver,
14 15 } from "./definitions.js";
16 +import { runObserver } from "./execution.js";
16 -const SUBCOMMANDS = ["list", "show", "validate"] as const;
18 +const SUBCOMMANDS = ["list", "show", "run", "validate"] as const;
17 19 const EMPTY: DiscoveryResult = {
18 20 observers: [],
19 21 diagnostics: [],
48 50 export default function witnessExtension(pi: ExtensionAPI): void {
49 51 let generation = 0;
52 + let lifecycleEpoch = 0;
50 53 let result = EMPTY;
51 54 let pending: Promise<DiscoveryResult> | undefined;
55 + let displayedResult:
56 + | { definition: ObserverDefinition; observerId: string }
57 + | undefined;
58 + let activeRun:
59 + | {
60 + controller: AbortController;
61 + definition: ObserverDefinition;
62 + observerId: string;
63 + sessionId: string;
64 + timer: ReturnType<typeof setTimeout>;
65 + }
66 + | undefined;
68 + const abortActiveRun = (reason: string): void => {
69 + const execution = activeRun;
70 + activeRun = undefined;
71 + if (!execution) return;
72 + clearTimeout(execution.timer);
73 + execution.controller.abort(new Error(reason));
74 + };
53 76 const startDiscovery = (
54 77 ctx: ExtensionContext,
58 81 const discovery = discoverObservers(observerRoots(ctx)).then((next) => {
59 82 if (currentGeneration !== generation) return result;
60 83 result = next;
84 + if (activeRun) {
85 + const replacement = next.observers.find(
86 + (observer) => observer.id === activeRun?.observerId,
87 + );
88 + if (
89 + replacement?.state !== "ready" ||
90 + !replacement.definition ||
91 + JSON.stringify(replacement.definition) !==
92 + JSON.stringify(activeRun.definition)
93 + ) {
94 + abortActiveRun(
95 + "Witness observer definition changed during execution",
96 + );
97 + ctx.ui.setWidget("witness", undefined);
98 + }
99 + }
100 + if (displayedResult) {
101 + const replacement = next.observers.find(
102 + (observer) => observer.id === displayedResult?.observerId,
103 + );
104 + if (
105 + replacement?.state !== "ready" ||
106 + !replacement.definition ||
107 + JSON.stringify(replacement.definition) !==
108 + JSON.stringify(displayedResult.definition)
109 + ) {
110 + displayedResult = undefined;
111 + ctx.ui.setWidget("witness", undefined);
61 114 if (warn && next.diagnostics.length > 0) {
62 115 ctx.ui.notify(
63 116 `Witness disabled ${next.observers.filter((observer) => observer.state !== "ready").length} observer(s) after ${next.diagnostics.length} validation diagnostic(s). Run /witness validate for details.`,
88 141 description: "Inspect and validate configured Witness observers",
89 142 getArgumentCompletions(prefix): AutocompleteItem[] | null {
90 143 const normalized = prefix.trimStart();
91 - const show = /^show\s+(\S*)$/i.exec(normalized);
92 - const items = show
144 + const observerCommand = /^(show|run)\s+(\S*)$/i.exec(normalized);
145 + const items = observerCommand
93 146 ? result.observers
94 - .filter((observer) => observer.id.startsWith(show[1] ?? ""))
147 + .filter((observer) =>
148 + observer.id.startsWith(observerCommand[2] ?? ""),
95 150 .map((observer) => ({
96 - value: `show ${observer.id}`,
151 + value: `${observerCommand[1]?.toLowerCase()} ${observer.id}`,
97 152 label: observer.id,
98 153 description: observer.state,
105 160 return items.length ? items : null;
107 162 async handler(args, ctx) {
163 + const commandEpoch = lifecycleEpoch;
108 164 const [command = "", id, ...extra] = args.trim().split(/\s+/);
110 166 !SUBCOMMANDS.some((candidate) => candidate === command) ||
111 167 extra.length > 0 ||
112 - (command === "show" && !id) ||
113 - (command !== "show" && id)
168 + (["show", "run"].includes(command) && !id) ||
169 + (!["show", "run"].includes(command) && id)
115 - notify(ctx, "Usage: /witness list | show <id> | validate", "error");
171 + notify(
172 + ctx,
173 + "Usage: /witness list | show <id> | run <id> | validate",
174 + "error",
175 + );
116 176 return;
118 178 let current: DiscoveryResult;
122 182 ? await startDiscovery(ctx, false)
123 183 : await currentResult(ctx);
124 184 } catch (error) {
185 + if (commandEpoch !== lifecycleEpoch) return;
125 186 notify(
127 188 `Witness definition discovery failed: ${error instanceof Error ? error.message : String(error)}`,
130 191 return;
193 + if (commandEpoch !== lifecycleEpoch) return;
132 194 if (command === "list") {
133 195 notify(
155 217 notify(ctx, `Unknown Witness observer: ${id}`, "error");
156 218 return;
158 - const lines = [formatObserver(observer)];
159 - if (observer.definition)
160 - lines.push(JSON.stringify(observer.definition, null, 2));
161 - if (observer.diagnostics.length)
162 - lines.push(...observer.diagnostics.map(formatDiagnostic));
163 - notify(
164 - ctx,
165 - lines.join("\n"),
166 - observer.state === "ready" ? "info" : "warning",
167 - );
220 + if (command === "show") {
221 + const lines = [formatObserver(observer)];
222 + if (observer.definition)
223 + lines.push(JSON.stringify(observer.definition, null, 2));
224 + if (observer.diagnostics.length)
225 + lines.push(...observer.diagnostics.map(formatDiagnostic));
226 + notify(
227 + ctx,
228 + lines.join("\n"),
229 + observer.state === "ready" ? "info" : "warning",
230 + );
231 + return;
233 + if (observer.state !== "ready" || !observer.definition) {
234 + notify(ctx, `Witness observer ${id} is ${observer.state}`, "error");
235 + return;
237 + if (activeRun) {
238 + notify(
239 + ctx,
240 + `Witness observer ${activeRun.observerId} is already running`,
241 + "warning",
242 + );
243 + return;
246 + const controller = new AbortController();
247 + const execution = {
248 + controller,
249 + definition: observer.definition,
250 + observerId: observer.id,
251 + sessionId: ctx.sessionManager.getSessionId(),
252 + timer: setTimeout(
253 + () =>
254 + controller.abort(
255 + new Error(
256 + `Witness observer ${observer.id} exceeded ${observer.definition?.agent.limits.timeoutMs} ms`,
257 + ),
258 + ),
259 + observer.definition.agent.limits.timeoutMs,
260 + ),
261 + };
262 + activeRun = execution;
263 + displayedResult = undefined;
264 + ctx.ui.setWidget("witness", [`Witness ${observer.id}: running`]);
265 + try {
266 + const output = await runObserver(
267 + observer.definition,
268 + ctx,
269 + controller.signal,
270 + );
271 + if (
272 + activeRun !== execution ||
273 + ctx.sessionManager.getSessionId() !== execution.sessionId
275 + return;
276 + displayedResult = {
277 + definition: observer.definition,
278 + observerId: observer.id,
279 + };
280 + ctx.ui.setWidget(
281 + "witness",
282 + output.text.split(/\r?\n/).map((line) => sanitize(line)),
283 + );
284 + } catch (error) {
285 + if (activeRun !== execution) return;
286 + const message = error instanceof Error ? error.message : String(error);
287 + displayedResult = undefined;
288 + ctx.ui.setWidget("witness", [sanitize(`Witness ${id}: failed`)]);
289 + notify(ctx, `Witness observer ${id} failed: ${message}`, "error");
290 + } finally {
291 + clearTimeout(execution.timer);
292 + if (activeRun === execution) activeRun = undefined;
183 - pi.on("session_shutdown", () => {
309 + const invalidateRun = (ctx: ExtensionContext, reason: string): void => {
310 + lifecycleEpoch++;
311 + abortActiveRun(reason);
312 + displayedResult = undefined;
313 + ctx.ui.setWidget("witness", undefined);
314 + };
315 + pi.on("session_before_switch", (_event, ctx) => {
316 + invalidateRun(ctx, "Witness execution invalidated by session switch");
317 + });
318 + pi.on("session_before_fork", (_event, ctx) => {
319 + invalidateRun(ctx, "Witness execution invalidated by session fork");
320 + });
321 + pi.on("session_before_tree", (_event, ctx) => {
322 + invalidateRun(ctx, "Witness execution invalidated by tree navigation");
323 + });
325 + pi.on("session_shutdown", (_event, ctx) => {
326 + invalidateRun(ctx, "Witness execution invalidated by session shutdown");
184 327 generation++;
185 328 pending = undefined;
186 329 result = EMPTY;

extensions/witness/package.json

Modified
7 7 "yaml": "^2.9.0"
9 9 "peerDependencies": {
10 + "@earendil-works/pi-ai": "*",
10 11 "@earendil-works/pi-coding-agent": "*",
11 12 "@earendil-works/pi-tui": "*"

package-lock.json

Modified
551 551 "vitest": "^4.1.9"
553 553 "peerDependencies": {
554 + "@earendil-works/pi-ai": "*",
554 555 "@earendil-works/pi-coding-agent": "*",
555 556 "@earendil-works/pi-tui": "*"