repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by admin

feat(witness): add definition discovery and inspection

3ae23c36fe693b8fe45f79a12e8cb4f52092e9bd

Oliver Krylow <o.krylow@isp-insoft.de> ·

SSH signature · not checked

Browse filesCompare with HEAD

Changed files (11)

CHANGELOG.md

Modified
2 2 # Changelog
4 4 ## Unreleased
6 +- Add Witness definition discovery and validation with trusted project overrides and `/witness list`, `/witness show`, and `/witness validate` commands
6 7 - Let IQ counter judging fall back to the next pool member when the assigned counter rejects a request (rate limits), rotating deterministically, since a lost agreement pair never returns once the session is scored; report exhausted pools separately as "counter pairs lost" instead of counting already-scored sessions as failed, and keep primary-judge failures failing so unscored sessions retry on the next scan
7 8 - Replace `/iq export` and its `iq.exportDir`/`iq.exportName` settings with `iq.stateDir`: one shared, synced directory for records, telemetry, and checks; the first use with an empty shared directory copies local state in once, records carry a `host` field for provenance, the leaderboard merges all hosts, and the scan lock moves to the OS temp directory so synced folders never carry lock files; unreadable IQ settings fall back to the local state directory with a warning instead of blocking commands
8 9 - Add `telemetry` and `checks` tabs to the IQ dashboard, bringing it to five tabs: telemetry shows measured per-model behavior (turns, tool calls, edit share, output volume, cache share, latency, speed) and checks shows each model's latest check result with task states, acceptance, estimate, and check-versus-history deltas

extensions/witness/PLAN.md

Added
new file mode 100644
1 +# Witness implementation plan
3 +## Goal
5 +Implement the Witness mechanism, then migrate existing policies one at a time without changing their proven behavior accidentally.
6 +Each slice ends with observable behavior and focused verification.
7 +Old extensions remain available until their replacement passes parity checks.
9 +## Entry conditions
11 +- [x] The `witness` name is selected.
12 +- [x] The extension scaffold loads through Pi's real extension runtime.
13 +- [x] [SPEC.md](SPEC.md) records the accepted design boundaries.
14 +- [x] Witness ships no observers or templates by default.
16 +## Slice 1: Load and explain definitions
18 +- [x] Parse one Markdown observer file with strict YAML frontmatter and a prompt body.
19 +- [x] Discover active user and trusted-project definitions asynchronously.
20 +- [x] Replace complete user definitions by matching project IDs without deep merge.
21 +- [x] Validate identity, trigger, context, agent, result, and delivery fields.
22 +- [x] Keep invalid and unresolved observers disabled.
23 +- [x] Add `/witness list`, `/witness show <id>`, and `/witness validate` with completions.
24 +- [x] Aggregate startup diagnostics into one warning while preserving exact command output.
25 +- [x] Test project trust, replacement semantics, malformed files, unknown fields, reload invalidation, and empty default discovery.
26 +- [x] Run `mise run //extensions/witness:test`.
28 +Exit condition: a user can inspect validated observer policy without running a model.
30 +## Slice 2: Run one manual text observer
32 +- [ ] Add `/witness run <id>` for a manually triggered text observer.
33 +- [ ] Resolve the configured model, authentication, thinking level, prompt, and bounds without silent fallback.
34 +- [ ] Create a minimal parent-linked child session in the parent's session store.
35 +- [ ] Prevent child resource inheritance and Witness recursion.
36 +- [ ] Persist real child assistant usage without copying usage into the parent.
37 +- [ ] Deliver sanitized text through the shared Witness widget.
38 +- [ ] Abort and invalidate children on timeout, disablement, reload, tree navigation, session replacement, and shutdown.
39 +- [ ] Test persistent and ephemeral parents, unavailable models, extension-registered providers, stale completions, and child-session attribution.
40 +- [ ] Compare warm and cold latency against direct `ctx.modelRegistry.complete()` on representative fast models.
41 +- [ ] Run `mise run //extensions/witness:test` and `mise run //extensions/witness:check`.
43 +Exit condition: one configured observer completes a real manual child run with visible output and accounted usage.
45 +## Slice 3: Migrate activity summaries
47 +- [ ] Add bounded normalized activity context with independent observer cursors.
48 +- [ ] Add activity debounce, maximum wait, immediate priority, final priority, and single-flight coalescing.
49 +- [ ] Keep persisted child history out of the bounded next-request context.
50 +- [ ] Add pause and resume behavior.
51 +- [ ] Create an `activity-summary` definition from the current TLDR policy without shipping it by default.
52 +- [ ] Preserve current sanitization, wrapping, tense, sparse-context behavior, timeout, token ceiling, and unavailable state.
53 +- [ ] Test `off` and the lowest supported thinking levels for latency and output quality on representative recorded activities.
54 +- [ ] Compare Witness output, latency, request count, and cost with the current TLDR extension.
55 +- [ ] Remove the TLDR extension only after parity and migration documentation pass.
56 +- [ ] Run `mise run //extensions/witness:test` and the smallest affected root check once.
58 +Exit condition: an installed activity-summary definition replaces TLDR without a perceptible speed regression.
60 +## Slice 4: Add templates and creation UI
62 +- [ ] Discover inactive user templates separately from active definitions.
63 +- [ ] Add `/witness add <template>` as an expanded standalone copy operation.
64 +- [ ] Add `/witness save-template <id>`.
65 +- [ ] Add `/witness create` with explanatory choices for triggers, context, models, reasoning, capabilities, results, effects, and attention.
66 +- [ ] Show the complete resolved definition and authority before writing.
67 +- [ ] Add `/witness open <id>`.
68 +- [ ] Document the mental model, configuration reference, safety model, troubleshooting, and copyable recipes.
69 +- [ ] Generate reference documentation from the same validation schema where practical.
70 +- [ ] Test cancelled creation, duplicate IDs, unavailable destinations, template independence, and global versus project targets.
71 +- [ ] Run `mise run //extensions/witness:test`.
73 +Exit condition: a human can create and understand an observer without hand-authoring frontmatter.
75 +## Slice 5: Migrate praised-work learning
77 +- [ ] Add exact and regular-expression user-input matching with explicit normalization.
78 +- [ ] Add immediate versus post-settlement timing and consume versus continue behavior.
79 +- [ ] Ignore extension-originated input and durably enqueue consumed jobs before handling input.
80 +- [ ] Add the schema-derived terminating `observer_report` tool.
81 +- [ ] Add validated structured records and atomic user archive collections.
82 +- [ ] Add `/witness records <collection>` with a generic compact and detailed record view.
83 +- [ ] Add restart recovery for queued and interrupted jobs.
84 +- [ ] Create a `praised-work-learning` definition from the current Good Job policy without shipping it by default.
85 +- [ ] Preserve exact trigger semantics, pinned source provenance, evidence requirements, bounded fields, notifications, and cross-project records.
86 +- [ ] Verify that enqueue failure lets consumed input continue to the parent.
87 +- [ ] Remove the Good Job extension only after record migration and recovery parity pass.
88 +- [ ] Run `mise run //extensions/witness:test` and the smallest affected root check once.
90 +Exit condition: configured praise triggers produce durable, browsable, usage-accounted learning records without starting a parent turn.
92 +## Slice 6: Migrate documentation synchronization
94 +- [ ] Add safe file, directory, and glob context sources.
95 +- [ ] Add workspace-mutation, quiet-window, cooldown, and periodic triggers.
96 +- [ ] Reconcile ambiguous Bash mutation through a bounded workspace fingerprint mechanism.
97 +- [ ] Add structured file-replacement reports restricted to supplied file snapshots.
98 +- [ ] Add duplicate-path rejection, symlink containment, snapshot hashing, and compare-and-swap application.
99 +- [ ] Add automatic, confirmed, and editor-reviewed application policies.
100 +- [ ] Create a `documentation-sync` definition from the current Pulse policy without shipping it by default.
101 +- [ ] Decide through measured use whether per-document model overrides justify their complexity.
102 +- [ ] Preserve manual execution, pause, status, no-op reporting, JSON correction behavior, and restart state where still useful.
103 +- [ ] Remove the Pulse extension only after file-safety and scheduling parity pass.
104 +- [ ] Run `mise run //extensions/witness:test` and the smallest affected root check once.
106 +Exit condition: a configured documentation-sync observer safely replaces Pulse's tracked-document workflow.
108 +## Slice 7: Migrate feedback learning
110 +- [ ] Add trigger-to-settled-run correlation from the recorded starting leaf.
111 +- [ ] Create a `feedback-learning` definition that matches configured `wtf` input and does not consume it.
112 +- [ ] Move immediate correction policy into an independent WTF skill.
113 +- [ ] Analyze the complaint, parent correction, tool evidence, and final outcome after settlement.
114 +- [ ] Archive one deduplicated evidence-backed lesson through configured delivery.
115 +- [ ] Remove learning instructions from the old WTF prompt after observer and skill behavior pass together.
116 +- [ ] Run `mise run //extensions/witness:test` and the relevant skill activation check.
118 +Exit condition: WTF feedback reaches the parent skill normally and produces a separate durable retrospective.
120 +## Slice 8: Add findings, questions, and attention brokering
122 +- [ ] Add stable findings, insufficient-context reports, and question reports.
123 +- [ ] Add the consolidated inbox with evidence, freshness, provenance, model, usage, dismiss, snooze, rerun, and answer actions.
124 +- [ ] Keep one compact shared widget instead of per-observer widgets.
125 +- [ ] Add explicitly configured automatic prompting with settlement, relevance, deduplication, cooldown, and single-dialog gates.
126 +- [ ] Ensure children never open parent dialogs directly.
127 +- [ ] Test competing observers, repeated findings, stale questions, dismissal, snooze, and queued parent activity.
128 +- [ ] Run `mise run //extensions/witness:test`.
130 +Exit condition: multiple observers share one quiet attention surface without prompt or widget spam.
132 +## Slice 9: Prove the extension as a complete replacement host
134 +- [ ] Exercise activity-summary, praised-work-learning, documentation-sync, and feedback-learning together.
135 +- [ ] Verify bounded concurrency with one live slot and one investigative slot under sustained activity.
136 +- [ ] Verify no observer or configured child extension can recursively load Witness.
137 +- [ ] Verify persisted usage is counted once by Pi session tooling and `ccusage`.
138 +- [ ] Measure startup context impact according to `docs/context-startup.md`.
139 +- [ ] Measure Windows and WSL startup, cold child creation, warm run latency, memory, session growth, and archive growth.
140 +- [ ] Update root documentation and release notes after obsolete extensions are removed.
141 +- [ ] Run root `mise run //:test` or `mise run //:check` once as the complete verification gate.
143 +Exit condition: Witness hosts migrated policies independently, accounts for their work, and remains quiet when no definitions are installed.
145 +## Deferred
147 +- Verification-audit policy remains deferred until migrated policies establish trustworthy evidence and attention mechanisms.
148 +- Arbitrary scripting, observer dependencies, observer-to-observer conversations, inherited parent resources, and custom renderer code remain out of scope.
149 +- Publishing a remote JSON Schema remains optional because runtime validation and the creation UI are authoritative.

extensions/witness/README.md

Added
new file mode 100644
1 +# witness
3 +Configurable background observers for Pi.
5 +## Install / load
7 +This extension is loaded through the root pi-ext package.
8 +See the [root README](../../README.md).
10 +## Current state
12 +Witness discovers and validates observer definitions without running models.
13 +It ships no observers or templates by default.
15 +Active user definitions live in `~/.pi/agent/witness/observers/`.
16 +Trusted project definitions live in `.pi/witness/` and completely replace matching user IDs.
18 +Use `/witness list`, `/witness show <id>`, and `/witness validate` to inspect the resolved configuration.
20 +The first slice accepts manual text observers with empty context and widget delivery:
22 +```markdown
23 +---
24 +id: review
25 +trigger:
26 + manual: true
27 +context:
28 + sources: []
29 +agent:
30 + model: provider/model
31 + thinking: low
32 + tools: []
33 + skills: []
34 + prompts: []
35 + extensions: []
36 +result:
37 + type: text
38 +delivery:
39 + type: widget
40 +---
42 +Review the supplied evidence.
43 +```
45 +Witness applies defaults of 30 seconds, 1,024 output tokens, zero retries, and zero tool calls under `agent.limits`.
46 +Unsupported future trigger, context, capability, result, and delivery forms remain disabled until their implementation slice lands.
47 +A `ready` definition has a catalogued model and valid slice-1 structure; authentication and execution readiness begin in slice 2.
49 +## Performance
51 +Run `mise run //extensions/witness:bench` from the repository root to benchmark 128-definition parsing and discovery.
52 +Run `mise run //extensions/witness:profile` to write a Node CPU profile under `.research/witness/`.
54 +The intended behavior is defined in [SPEC.md](SPEC.md).
55 +Implementation and migrations are ordered in [PLAN.md](PLAN.md).

extensions/witness/SPEC.md

Added
new file mode 100644
1 +# Witness specification
3 +## Status
5 +This document defines the intended Witness behavior.
6 +Implementation slice 1 discovers, validates, and explains definitions without running models.
8 +## Purpose
10 +Witness runs user-configured background agents in response to Pi activity.
11 +Witness provides mechanism, while observer definitions provide policy.
12 +Witness ships no enabled observers and no templates by default.
14 +## Boundaries
16 +Witness does not replace Pi prompts, skills, commands, model selection, or the parent agent loop.
17 +Witness does not contain documentation, praise, feedback, verification, or activity-summary policy.
18 +Witness does not depend on another pi-ext extension.
19 +Witness may load extensions explicitly named by an observer definition.
20 +Witness must not load itself into a child runtime.
22 +## Observer definition
24 +One observer is defined by one Markdown file with YAML frontmatter.
25 +The Markdown body is the observer system prompt.
26 +Frontmatter defines identity, triggers, context, child-agent configuration, result protocol, and delivery.
27 +Referenced files are resolved relative to the observer file unless a field explicitly defines another base.
28 +Unknown fields are validation errors.
29 +Invalid observers never partially activate.
31 +Active user observers live below the Pi agent directory in `witness/observers/`.
32 +Active project observers live in `.pi/witness/`.
33 +Reusable user templates live below the Pi agent directory in `witness/templates/`.
34 +Project definitions are read only when the project is trusted.
35 +A project observer with the same ID replaces the complete user observer definition.
36 +Definitions never deep-merge across scopes.
38 +## Identity and templates
40 +Observer IDs are stable, unique within the resolved configuration, and safe as file-name components.
41 +A template is an inactive observer definition available to the creation UI.
42 +`/witness add <template>` copies an expanded standalone definition into an active scope.
43 +An installed observer retains no inheritance link to its template.
44 +`/witness save-template <observer-id>` copies an active observer into the user template library.
45 +`/witness create` may create an active observer, a reusable template, or both.
47 +## Triggers
49 +Witness supports these trigger mechanisms:
51 +- Parent activity with debounce and maximum-wait bounds.
52 +- Parent settlement after all automatic retries, compaction retries, and queued continuation finish.
53 +- Successful observed workspace mutation followed by a quiet period.
54 +- A periodic interval.
55 +- Explicit manual execution.
56 +- Deterministic matching of raw user input.
58 +Input matching supports exact normalized values and configured regular expressions.
59 +Input triggers ignore extension-originated messages.
60 +An input trigger chooses immediate execution or execution after the resulting parent run settles.
61 +An input trigger chooses whether the matching message continues to the parent agent.
62 +Witness persists a consumed input job before returning `handled`.
63 +If durable enqueue fails, the input continues to the parent agent instead of being lost.
64 +A post-settlement input trigger records the trigger text and starting parent leaf before the parent run starts.
66 +Triggers schedule work but do not grant context or capabilities.
67 +Repeated triggers coalesce according to observer policy.
68 +Each observer has at most one running job and one replaceable pending job.
69 +Continuous activity cannot postpone a configured observer forever when a maximum wait is set.
70 +No observer runs merely because time passed when its configured evidence has not changed, except an explicit periodic trigger.
72 +## Context
74 +Witness builds bounded context from explicitly configured sources.
75 +Supported sources include normalized parent activity, active-branch conversation, selected files, file globs, and observed tool execution evidence.
76 +Every source has an explicit or mechanical size bound.
77 +Context records preserve stable evidence IDs where later results may cite evidence.
78 +Truncation and unavailable sources are disclosed to the child.
79 +Missing evidence must not be represented as evidence of absence.
80 +All parent messages, files, tool inputs, and tool results are untrusted data inside observer prompts.
81 +One observer consuming a checkpoint never discards another observer's evidence.
82 +Persisted child transcripts do not determine the next model context unless the observer explicitly selects them.
84 +## Child agents
86 +Every model-backed run uses a Pi child session.
87 +A persistent parent produces a persistent child in the same session store with a `parentSession` link.
88 +An ephemeral parent may use an in-memory child.
89 +Actual child assistant messages retain their provider, model, token usage, and cost.
90 +Child usage is never copied onto parent assistant messages.
92 +An observer explicitly selects its model and thinking level.
93 +Witness does not silently substitute another model or thinking level.
94 +An observer explicitly selects skills, tools, prompt resources, and extensions.
95 +No parent skill, tool, prompt, context file, or extension is inherited implicitly.
96 +Tools are restricted by a construction-time allowlist.
97 +The child cannot activate Witness or recursively create observers.
99 +A configured extension is trusted executable code with the child's host-process authority.
100 +Listing an extension in trusted configuration is the authorization to load it.
101 +Witness loads exactly the listed extensions and reports load failures.
102 +Witness canonicalizes package identity and real paths when excluding itself, including aliases and symlinks.
103 +Tool restrictions do not constrain direct code executed by a configured extension.
105 +Observer policy may bound wall time, output tokens, retries, tool calls, and aggregate usage.
106 +Witness aborts work that exceeds configured bounds.
107 +A stale child result cannot affect a replaced session, branch, observer definition, or disabled observer.
108 +A slower result may remain visible as explicitly stale while newer relevant evidence waits for reconciliation.
110 +## Results
112 +A text observer returns its final assistant text directly.
113 +Text observers do not receive a submission tool merely for uniformity.
114 +Structured observers receive one mechanism-owned terminating `observer_report` tool.
115 +The report schema is derived from the configured result type.
116 +The report tool validates and captures data but performs no external effect.
117 +Malformed structured output cannot reach a delivery effect.
119 +Supported result families are text, findings, questions, records, and file replacements.
120 +A finding has a stable identity, concise claim, evidence references, severity, and optional suggested action.
121 +A question has a stable identity, concise question, reason, evidence references, and optional choices.
122 +A record is validated structured data stored in a named collection.
123 +A file replacement names an allowed input file, complete replacement content, and a concise summary.
124 +An observer may explicitly report insufficient context without producing a finding or effect.
126 +## Delivery and effects
128 +Witness owns delivery after result validation.
129 +Supported delivery mechanisms are the shared widget, inbox, notification, archive, and controlled file application.
130 +Observer agents propose effects but never execute delivery effects through `observer_report`.
131 +Investigative tools granted by policy remain ordinary child capabilities and are independent of delivery.
133 +Archive collections support user, session, project, and explicit-path storage policies.
134 +User archives use the platform application-data directory below `pi/witness/records/<collection>/`.
135 +Session archives use parent custom entries and do not enter model context.
136 +Project and explicit-path archives require trusted configuration.
137 +Archive writes are atomic and validated before replacement.
139 +File application is restricted to files supplied through the run's configured file context.
140 +Witness rejects unknown paths, duplicate paths, changed snapshots, and symlink escapes.
141 +Application policy is `auto`, `confirm`, or `review`.
142 +Confirmation and review happen in the parent UI after the child has finished.
143 +Cancellation leaves files unchanged.
145 +## Attention and UI
147 +Witness owns one shared compact widget for all observers.
148 +Observers do not create independent widgets.
149 +The widget may combine one transient status with aggregate finding and question counts.
150 +One consolidated review surface exposes observer status, findings, questions, evidence, freshness, child-session provenance, and usage.
151 +Stable result IDs update existing items instead of generating repeated alerts.
152 +Dismissal and snooze state survive reconstruction according to configured scope.
153 +Unavailable, incomplete, stale, clear, and disabled are distinct states.
155 +Child agents and child extensions do not open parent dialogs directly.
156 +A question first enters the Witness inbox.
157 +Automatic prompting is disabled unless observer policy enables it.
158 +An automatic prompt may appear only after the parent settles, while no other Witness prompt is active, while the question remains current, and while the configured global attention cooldown permits it.
159 +Witness presents at most one automatic question at a time.
160 +Answering a question does not authorize its suggested effect.
162 +## Commands
164 +Witness eventually provides these commands:
166 +- `/witness list` lists resolved observers and their state.
167 +- `/witness show <id>` shows the resolved policy, defaults, capabilities, and authority.
168 +- `/witness create` opens the explanatory creation wizard.
169 +- `/witness add <template>` installs a standalone copy of a template.
170 +- `/witness save-template <id>` saves an observer as a reusable user template.
171 +- `/witness validate` validates every discovered definition and resource reference.
172 +- `/witness run <id>` manually executes an observer.
173 +- `/witness pause <id>` pauses one observer.
174 +- `/witness resume <id>` resumes one observer.
175 +- `/witness open <id>` opens an observer definition.
176 +- `/witness records <collection>` browses archived records.
178 +Commands with arguments provide completions.
179 +The creation wizard explains each option tersely, including context exposure, cost, latency, executable extensions, mutation authority, and interruption policy.
180 +The wizard displays the complete resolved definition before writing it.
182 +## Validation and startup
184 +Witness registers its fixed Pi hooks and commands without waiting for filesystem, process, or network work.
185 +Definition discovery and validation run asynchronously after session start.
186 +Configured observers remain disabled until their current definitions validate.
187 +One aggregated warning reports invalid definitions after background validation.
188 +`/witness validate` reports exact files, fields, and reasons.
189 +Structural validity, unavailable resources, and unavailable authentication are distinct diagnostics.
190 +Validation does not instantiate child agents or contact providers.
191 +Reload, session replacement, and shutdown invalidate unfinished discovery and validation work.
193 +## Safety and observability
195 +Witness sanitizes all text before terminal display.
196 +Witness never prints credentials, secrets, raw private context, or child prompts in routine diagnostics.
197 +Filesystem sources and effects reject path escapes and external symlink targets unless an explicit trusted-path policy authorizes them.
198 +Every effect records the observer ID, triggering boundary, child session, model, usage, result identity, and outcome.
199 +Witness provides enough provenance to explain why an observer ran and what evidence it saw.
201 +## Initial distribution
203 +Witness ships with no observers or templates enabled or installed.
204 +Activity summaries, documentation synchronization, praised-work learning, feedback learning, and verification auditing are migration targets, not built-in policy.

extensions/witness/__bench__/definitions.bench.ts

Added
new file mode 100644
1 +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
2 +import { cpus, tmpdir } from "node:os";
3 +import { join } from "node:path";
4 +import { discoverObservers, parseObserverDefinition } from "../definitions.ts";
6 +const OBSERVER_COUNT = 128;
7 +const ROUNDS = 12;
8 +const MINIMUM_ROUND_NS = 200_000_000;
10 +function source(index: number): string {
11 + return `---
12 +id: observer-${index.toString().padStart(4, "0")}
13 +description: Benchmark observer ${index}
14 +trigger:
15 + manual: true
16 +context:
17 + sources: []
18 +agent:
19 + model: test/model
20 + thinking: low
21 + tools: []
22 + skills: []
23 + prompts: []
24 + extensions: []
25 + limits:
26 + timeoutMs: 30000
27 + outputTokens: 1024
28 + retries: 0
29 + toolCalls: 0
30 +result:
31 + type: text
32 +delivery:
33 + type: widget
34 +---
36 +Inspect benchmark evidence ${index}.
37 +`;
40 +const sources = Array.from({ length: OBSERVER_COUNT }, (_, index) =>
41 + source(index),
42 +);
43 +let sink = 0;
45 +async function benchmark(
46 + name: string,
47 + operation: () => void | Promise<void>,
48 +): Promise<void> {
49 + for (let index = 0; index < 3; index++) await operation();
50 + let iterations = 1;
51 + while (true) {
52 + const started = process.hrtime.bigint();
53 + for (let index = 0; index < iterations; index++) await operation();
54 + if (Number(process.hrtime.bigint() - started) >= MINIMUM_ROUND_NS) break;
55 + iterations *= 2;
56 + }
57 + for (let round = 0; round < ROUNDS; round++) {
58 + const started = process.hrtime.bigint();
59 + for (let index = 0; index < iterations; index++) await operation();
60 + const elapsed = Number(process.hrtime.bigint() - started);
61 + console.log(
62 + `Benchmark${name}-${cpus().length}\t${iterations}\t${(elapsed / iterations).toFixed(0)} ns/op`,
63 + );
64 + }
67 +async function main(): Promise<void> {
68 + const root = await mkdtemp(join(tmpdir(), "witness-bench-"));
69 + const userRoot = join(root, "user");
70 + const projectRoot = join(root, "project");
71 + await mkdir(userRoot);
72 + await Promise.all(
73 + sources.map((definition, index) =>
74 + writeFile(
75 + join(userRoot, `${index.toString().padStart(4, "0")}.md`),
76 + definition,
77 + ),
78 + ),
79 + );
80 + const ctx = {
81 + isProjectTrusted: () => true,
82 + modelRegistry: {
83 + find: (provider: string, model: string) =>
84 + provider === "test" && model === "model"
85 + ? { provider, id: model }
86 + : undefined,
87 + },
88 + } as never;
90 + try {
91 + console.log(`goos: ${process.platform}`);
92 + console.log(`goarch: ${process.arch}`);
93 + console.log("pkg: pi-ext/witness");
94 + await benchmark(`ParseCatalog${OBSERVER_COUNT}`, () => {
95 + for (const definition of sources)
96 + sink += parseObserverDefinition(definition).id.length;
97 + });
98 + await benchmark(`DiscoverCatalog${OBSERVER_COUNT}`, async () => {
99 + const result = await discoverObservers({ userRoot, projectRoot, ctx });
100 + if (result.observers.length !== OBSERVER_COUNT)
101 + throw new Error("discovery benchmark returned an invalid result");
102 + sink += result.observers.length;
103 + });
104 + } finally {
105 + await rm(root, { recursive: true, force: true });
109 +await main();
110 +if (sink === 0) throw new Error("benchmark result was not consumed");

extensions/witness/__tests__/harness.test.ts

Added
new file mode 100644
1 +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
2 +import { tmpdir } from "node:os";
3 +import path from "node:path";
4 +import { afterEach, describe, expect, it, vi } from "vitest";
5 +import {
6 + createTestSession,
7 + type TestSession,
8 + withProcessEnv,
9 +} from "../../../test/harness";
10 +import witnessExtension, {
11 + discoverObservers,
12 + parseObserverDefinition,
13 +} from "../index";
15 +const observer = (
16 + options: {
17 + id?: string;
18 + model?: string;
19 + prompt?: string;
20 + extra?: string;
21 + } = {},
22 +) => `---
23 +id: ${options.id ?? "review"}
24 +trigger:
25 + manual: true
26 +context:
27 + sources: []
28 +agent:
29 + model: ${options.model ?? "test/model"}
30 + thinking: low
31 + tools: []
32 + skills: []
33 + prompts: []
34 + extensions: []
35 +result:
36 + type: text
37 +delivery:
38 + type: widget
39 +${options.extra ?? ""}---
41 +${options.prompt ?? "Review the supplied evidence."}
42 +`;
44 +function context(cwd: string, trusted: boolean, models = ["test/model"]) {
45 + return {
46 + cwd,
47 + isProjectTrusted: () => trusted,
48 + modelRegistry: {
49 + find: (provider: string, model: string) =>
50 + models.includes(`${provider}/${model}`)
51 + ? { provider, id: model }
52 + : undefined,
53 + },
54 + } as never;
57 +async function fixture() {
58 + const root = await mkdtemp(path.join(tmpdir(), "witness-"));
59 + const userRoot = path.join(root, "user");
60 + const projectRoot = path.join(root, "project", ".pi", "witness");
61 + await Promise.all([
62 + mkdir(userRoot, { recursive: true }),
63 + mkdir(projectRoot, { recursive: true }),
64 + ]);
65 + return {
66 + root,
67 + userRoot,
68 + projectRoot,
69 + cwd: path.dirname(path.dirname(projectRoot)),
70 + };
73 +describe("witness definitions", () => {
74 + const roots: string[] = [];
75 + afterEach(async () => {
76 + await Promise.all(
77 + roots.splice(0).map((root) => rm(root, { recursive: true, force: true })),
78 + );
79 + });
81 + it("parses strict frontmatter, applies mechanical defaults, and keeps the prompt body", () => {
82 + const parsed = parseObserverDefinition(observer());
83 + expect(parsed).toMatchObject({
84 + id: "review",
85 + trigger: { manual: true },
86 + context: { sources: [] },
87 + agent: {
88 + model: "test/model",
89 + thinking: "low",
90 + limits: {
91 + timeoutMs: 30_000,
92 + outputTokens: 1_024,
93 + retries: 0,
94 + toolCalls: 0,
95 + },
96 + },
97 + result: { type: "text" },
98 + delivery: { type: "widget" },
99 + prompt: "Review the supplied evidence.",
100 + });
101 + });
103 + it.each([
104 + ["missing frontmatter", "# prompt", "frontmatter"],
106 + "duplicate YAML key",
107 + observer().replace("id: review", "id: review\nid: again"),
108 + "frontmatter",
109 + ],
111 + "empty body",
112 + observer().replace("Review the supplied evidence.", ""),
113 + "prompt",
114 + ],
115 + ["invalid id", observer({ id: "Bad ID" }), "id"],
117 + "YAML alias",
118 + observer()
119 + .replace("id: review", "id: &identity review")
120 + .replace("model: test/model", "model: *identity"),
121 + "frontmatter",
122 + ],
124 + "unknown nested field",
125 + observer().replace(" manual: true", " manual: true\n surprise: true"),
126 + "trigger.surprise",
127 + ],
129 + "unknown top-level field",
130 + observer({ extra: "surprise: true\n" }),
131 + "surprise",
132 + ],
133 + ])("rejects %s", (_name, source, field) => {
134 + expect(() => parseObserverDefinition(source)).toThrow();
135 + try {
136 + parseObserverDefinition(source);
137 + } catch (error) {
138 + expect(error).toMatchObject({ field });
140 + });
142 + it("keeps source locations in malformed YAML diagnostics", () => {
143 + expect(() =>
144 + parseObserverDefinition("---\nid: one\nid: two\n---\nprompt\n"),
145 + ).toThrow("at line 2, column 1");
146 + });
148 + it("retains a valid candidate id on later validation errors", () => {
149 + const source = observer({ id: "broken" }).replace(
150 + " manual: true",
151 + " manual: nope",
152 + );
153 + try {
154 + parseObserverDefinition(source);
155 + expect.fail("expected validation failure");
156 + } catch (error) {
157 + expect(error).toMatchObject({ id: "broken", field: "trigger.manual" });
159 + });
161 + it.each(["tools", "skills", "prompts", "extensions"])(
162 + "keeps observers with slice-2 %s capabilities disabled",
163 + (field) => {
164 + const source = observer().replace(
165 + ` ${field}: []`,
166 + ` ${field}: [example]`,
167 + );
168 + expect(() => parseObserverDefinition(source)).toThrow(
169 + "must be empty because child capabilities start in slice 2",
170 + );
171 + try {
172 + parseObserverDefinition(source);
173 + } catch (error) {
174 + expect(error).toMatchObject({ field: `agent.${field}` });
176 + },
177 + );
179 + it("ignores project definitions until the project is trusted", async () => {
180 + const f = await fixture();
181 + roots.push(f.root);
182 + await writeFile(
183 + path.join(f.projectRoot, "project.md"),
184 + observer({ id: "project" }),
185 + );
187 + const untrusted = await discoverObservers({
188 + userRoot: f.userRoot,
189 + projectRoot: f.projectRoot,
190 + ctx: context(f.cwd, false),
191 + });
192 + const trusted = await discoverObservers({
193 + userRoot: f.userRoot,
194 + projectRoot: f.projectRoot,
195 + ctx: context(f.cwd, true),
196 + });
198 + expect(untrusted.observers).toEqual([]);
199 + expect(trusted.observers.map((item) => item.id)).toEqual(["project"]);
200 + });
202 + it("replaces the complete user definition with the matching project id", async () => {
203 + const f = await fixture();
204 + roots.push(f.root);
205 + await writeFile(
206 + path.join(f.userRoot, "review.md"),
207 + observer({ prompt: "user prompt" }),
208 + );
209 + await writeFile(
210 + path.join(f.projectRoot, "review.md"),
211 + observer({ prompt: "project prompt" }),
212 + );
214 + const result = await discoverObservers({
215 + userRoot: f.userRoot,
216 + projectRoot: f.projectRoot,
217 + ctx: context(f.cwd, true),
218 + });
220 + expect(result.observers).toHaveLength(1);
221 + expect(result.observers[0]).toMatchObject({
222 + scope: "project",
223 + state: "ready",
224 + definition: { prompt: "project prompt" },
225 + });
226 + expect(result.observers[0]?.replaces).toBe(
227 + path.join(f.userRoot, "review.md"),
228 + );
229 + });
231 + it("lets an invalid project replacement disable the matching user observer", async () => {
232 + const f = await fixture();
233 + roots.push(f.root);
234 + await writeFile(
235 + path.join(f.userRoot, "review.md"),
236 + observer({ prompt: "valid user" }),
237 + );
238 + await writeFile(
239 + path.join(f.projectRoot, "review.md"),
240 + observer().replace(
241 + "delivery:\n type: widget",
242 + "delivery:\n type: archive",
243 + ),
244 + );
246 + const result = await discoverObservers({
247 + userRoot: f.userRoot,
248 + projectRoot: f.projectRoot,
249 + ctx: context(f.cwd, true),
250 + });
252 + expect(result.observers).toMatchObject([
253 + { id: "review", scope: "project", state: "invalid" },
254 + ]);
255 + });
257 + it("keeps malformed, duplicate, and unresolved definitions disabled with exact diagnostics", async () => {
258 + const f = await fixture();
259 + roots.push(f.root);
260 + await writeFile(
261 + path.join(f.userRoot, "bad.md"),
262 + observer({ id: "bad" }).replace(" manual: true", " manual: nope"),
263 + );
264 + await writeFile(
265 + path.join(f.userRoot, "one.md"),
266 + observer({ id: "duplicate" }),
267 + );
268 + await writeFile(
269 + path.join(f.userRoot, "two.md"),
270 + observer({ id: "duplicate" }),
271 + );
272 + await writeFile(
273 + path.join(f.userRoot, "missing-model.md"),
274 + observer({ id: "missing-model", model: "missing/model" }),
275 + );
277 + const result = await discoverObservers({
278 + userRoot: f.userRoot,
279 + projectRoot: f.projectRoot,
280 + ctx: context(f.cwd, true),
281 + });
283 + expect(result.observers.every((item) => item.state !== "ready")).toBe(true);
284 + const duplicateDiagnostics = result.diagnostics.filter(
285 + (diagnostic) => diagnostic.id === "duplicate",
286 + );
287 + expect(
288 + duplicateDiagnostics.map((diagnostic) => diagnostic.file).sort(),
289 + ).toEqual(
290 + [path.join(f.userRoot, "one.md"), path.join(f.userRoot, "two.md")].sort(),
291 + );
292 + expect(result.diagnostics).toEqual(
293 + expect.arrayContaining([
294 + expect.objectContaining({
295 + id: "bad",
296 + field: "trigger.manual",
297 + kind: "invalid",
298 + }),
299 + expect.objectContaining({
300 + id: "duplicate",
301 + field: "id",
302 + message: 'duplicate user observer id "duplicate"',
303 + }),
304 + expect.objectContaining({
305 + id: "missing-model",
306 + field: "agent.model",
307 + kind: "unavailable",
308 + message: 'model "missing/model" is unavailable',
309 + }),
310 + ]),
311 + );
312 + });
314 + it("discovers no definitions from empty default directories", async () => {
315 + const f = await fixture();
316 + roots.push(f.root);
317 + await rm(f.userRoot, { recursive: true });
318 + await rm(f.projectRoot, { recursive: true });
319 + await expect(
320 + discoverObservers({
321 + userRoot: f.userRoot,
322 + projectRoot: f.projectRoot,
323 + ctx: context(f.cwd, true),
324 + }),
325 + ).resolves.toMatchObject({ observers: [], diagnostics: [] });
326 + });
327 +});
329 +describe("witness pi-test-harness", () => {
330 + let t: TestSession | undefined;
332 + afterEach(() => {
333 + t?.dispose();
334 + t = undefined;
335 + });
337 + it("loads through Pi and registers one command with completions", async () => {
338 + t = await createTestSession({ extensionFactories: [witnessExtension] });
339 + const extension = t.session.extensionRunner.extensions[0];
340 + const command = extension?.commands.get("witness");
342 + expect(t.session.extensionRunner.extensions).toHaveLength(1);
343 + expect([...extension.commands.keys()]).toEqual(["witness"]);
344 + expect(command?.getArgumentCompletions?.("sh")).toEqual([
345 + { value: "show", label: "show" },
346 + ]);
347 + });
349 + it("validates, lists, shows, and reloads current definitions", async () => {
350 + t = await createTestSession({ extensionFactories: [witnessExtension] });
351 + const home = t.env.HOME as string;
352 + const definitions = path.join(home, ".pi", "agent", "witness", "observers");
353 + await mkdir(definitions, { recursive: true });
354 + const runner = t.session.extensionRunner;
355 + const ctx = runner.createCommandContext();
356 + const model = ctx.modelRegistry.getAvailable()[0];
357 + expect(model).toBeDefined();
358 + await writeFile(
359 + path.join(definitions, "review.md"),
360 + observer({ model: `${model?.provider}/${model?.id}` }),
361 + );
362 + const command = runner.extensions[0]?.commands.get("witness");
363 + const notify = vi.spyOn(ctx.ui, "notify");
365 + await withProcessEnv(t.env, () => command?.handler("validate", ctx));
366 + expect(notify).toHaveBeenLastCalledWith(
367 + expect.stringContaining("Valid: 1 observer(s)."),
368 + "info",
369 + );
370 + expect(command?.getArgumentCompletions?.("show r")).toEqual([
371 + expect.objectContaining({ value: "show review" }),
372 + ]);
374 + await withProcessEnv(t.env, () => command?.handler("show review", ctx));
375 + expect(notify).toHaveBeenLastCalledWith(
376 + expect.stringContaining('"prompt": "Review the supplied evidence."'),
377 + "info",
378 + );
379 + await withProcessEnv(t.env, () => command?.handler("list", ctx));
380 + expect(notify).toHaveBeenLastCalledWith(
381 + expect.stringContaining("review: ready"),
382 + "info",
383 + );
385 + await writeFile(
386 + path.join(definitions, "review.md"),
387 + observer({ model: `${model?.provider}/${model?.id}` }).replace(
388 + " manual: true",
389 + " manual: broken",
390 + ),
391 + );
392 + await withProcessEnv(t.env, () => command?.handler("validate", ctx));
393 + expect(notify).toHaveBeenLastCalledWith(
394 + expect.stringContaining("trigger.manual: invalid"),
395 + "warning",
396 + );
397 + });
399 + it("reports a discovery failure and permits a clean retry", async () => {
400 + t = await createTestSession({ extensionFactories: [witnessExtension] });
401 + const definitions = path.join(
402 + t.env.HOME as string,
403 + ".pi",
404 + "agent",
405 + "witness",
406 + "observers",
407 + );
408 + await mkdir(definitions, { recursive: true });
409 + const runner = t.session.extensionRunner;
410 + const ctx = runner.createCommandContext();
411 + const model = ctx.modelRegistry.getAvailable()[0];
412 + await writeFile(
413 + path.join(definitions, "review.md"),
414 + observer({ model: `${model?.provider}/${model?.id}` }),
415 + );
416 + const command = runner.extensions[0]?.commands.get("witness");
417 + const notify = vi.spyOn(ctx.ui, "notify");
418 + vi.spyOn(ctx.modelRegistry, "find")
419 + .mockImplementationOnce(() => {
420 + throw new Error("catalog failed");
421 + })
422 + .mockImplementation((provider, id) =>
423 + ctx.modelRegistry
424 + .getAvailable()
425 + .find(
426 + (candidate) =>
427 + candidate.provider === provider && candidate.id === id,
428 + ),
429 + );
431 + await expect(
432 + withProcessEnv(t.env, () => command?.handler("validate", ctx)),
433 + ).resolves.toBeUndefined();
434 + expect(notify).toHaveBeenLastCalledWith(
435 + expect.stringContaining("catalog failed"),
436 + "error",
437 + );
439 + await withProcessEnv(t.env, () => command?.handler("validate", ctx));
440 + expect(notify).toHaveBeenLastCalledWith(
441 + expect.stringContaining("Valid: 1 observer(s)."),
442 + "info",
443 + );
444 + });
446 + it("aggregates startup diagnostics into one warning", async () => {
447 + const root = await mkdtemp(path.join(tmpdir(), "witness-startup-"));
448 + const definitions = path.join(
449 + root,
450 + ".test-home",
451 + ".pi",
452 + "agent",
453 + "witness",
454 + "observers",
455 + );
456 + await mkdir(definitions, { recursive: true });
457 + await writeFile(path.join(definitions, "one.md"), "not frontmatter");
458 + await writeFile(path.join(definitions, "two.md"), "also not frontmatter");
459 + try {
460 + t = await createTestSession({
461 + cwd: root,
462 + extensionFactories: [witnessExtension],
463 + });
464 + await expect
465 + .poll(
466 + () =>
467 + t?.events
468 + .uiCallsFor("notify")
469 + .filter((call) => call.args[1] === "warning") ?? [],
471 + .toHaveLength(1);
472 + expect(t.events.uiCallsFor("notify")[0]?.args[0]).toContain(
473 + "2 validation diagnostic(s)",
474 + );
475 + } finally {
476 + t?.dispose();
477 + t = undefined;
478 + await rm(root, { recursive: true, force: true });
480 + });
482 + it("invalidates discovered state on shutdown", async () => {
483 + t = await createTestSession({ extensionFactories: [witnessExtension] });
484 + const definitions = path.join(
485 + t.env.HOME as string,
486 + ".pi",
487 + "agent",
488 + "witness",
489 + "observers",
490 + );
491 + await mkdir(definitions, { recursive: true });
492 + const runner = t.session.extensionRunner;
493 + const ctx = runner.createCommandContext();
494 + const model = ctx.modelRegistry.getAvailable()[0];
495 + await writeFile(
496 + path.join(definitions, "review.md"),
497 + observer({ model: `${model?.provider}/${model?.id}` }),
498 + );
499 + const command = runner.extensions[0]?.commands.get("witness");
500 + await withProcessEnv(t.env, () => command?.handler("validate", ctx));
501 + expect(command?.getArgumentCompletions?.("show r")).not.toBeNull();
503 + await runner.emit({ type: "session_shutdown", reason: "reload" });
505 + expect(command?.getArgumentCompletions?.("show r")).toBeNull();
506 + });
507 +});

extensions/witness/definitions.ts

Added
new file mode 100644
1 +import type { Dirent } from "node:fs";
2 +import { readdir, readFile } from "node:fs/promises";
3 +import { join, relative } from "node:path";
4 +import type { ExtensionContext } from "@earendil-works/pi-coding-agent";
5 +import { isAlias, isMap, isSeq, type Node, parseDocument } from "yaml";
7 +export type ObserverScope = "user" | "project";
8 +export type DiagnosticKind = "invalid" | "unavailable";
10 +export interface WitnessDiagnostic {
11 + file: string;
12 + field: string;
13 + kind: DiagnosticKind;
14 + message: string;
15 + id?: string;
18 +export interface ObserverDefinition {
19 + id: string;
20 + description?: string;
21 + trigger: { manual: true };
22 + context: { sources: readonly [] };
23 + agent: {
24 + model: string;
25 + thinking: "off" | "minimal" | "low" | "medium" | "high" | "xhigh" | "max";
26 + tools: readonly string[];
27 + skills: readonly string[];
28 + prompts: readonly string[];
29 + extensions: readonly string[];
30 + limits: {
31 + timeoutMs: number;
32 + outputTokens: number;
33 + retries: number;
34 + toolCalls: number;
35 + };
36 + };
37 + result: { type: "text" };
38 + delivery: { type: "widget" };
39 + prompt: string;
42 +export interface ResolvedObserver {
43 + definition?: ObserverDefinition;
44 + id: string;
45 + file: string;
46 + scope: ObserverScope;
47 + state: "ready" | "invalid" | "unavailable";
48 + diagnostics: readonly WitnessDiagnostic[];
49 + replaces?: string;
52 +export interface DiscoveryResult {
53 + observers: readonly ResolvedObserver[];
54 + diagnostics: readonly WitnessDiagnostic[];
55 + projectTrusted: boolean;
58 +const DEFAULT_LIMITS = {
59 + timeoutMs: 30_000,
60 + outputTokens: 1_024,
61 + retries: 0,
62 + toolCalls: 0,
63 +} as const;
64 +const ID = /^[a-z0-9](?:[a-z0-9-]{0,62}[a-z0-9])?$/;
65 +const THINKING = new Set([
66 + "off",
67 + "minimal",
68 + "low",
69 + "medium",
70 + "high",
71 + "xhigh",
72 + "max",
73 +]);
74 +const TOP_LEVEL = new Set([
75 + "id",
76 + "description",
77 + "trigger",
78 + "context",
79 + "agent",
80 + "result",
81 + "delivery",
82 +]);
83 +class DefinitionError extends Error {
84 + readonly field: string;
85 + readonly id?: string;
87 + constructor(field: string, message: string, id?: string) {
88 + super(message);
89 + this.field = field;
90 + this.id = id;
91 + }
94 +function fail(field: string, message: string): never {
95 + throw new DefinitionError(field, message);
98 +function rejectAliases(node: Node | null): void {
99 + if (!node) return;
100 + if (isAlias(node)) fail("frontmatter", "YAML aliases are not allowed");
101 + if (isSeq(node))
102 + for (const item of node.items) rejectAliases(item as Node | null);
103 + if (isMap(node)) {
104 + for (const pair of node.items) {
105 + rejectAliases(pair.key as Node | null);
106 + rejectAliases(pair.value as Node | null);
111 +function splitDocument(source: string): { yaml: string; prompt: string } {
112 + const match = /^---[\t ]*\r?\n([\s\S]*?)\r?\n---[\t ]*(?:\r?\n|$)/.exec(
113 + source,
114 + );
115 + if (!match)
116 + fail("frontmatter", "expected one leading YAML frontmatter block");
117 + const prompt = source.slice(match[0].length).trim();
118 + if (!prompt) fail("prompt", "Markdown prompt body must not be empty");
119 + return { yaml: match[1] ?? "", prompt };
122 +function record(value: unknown, field: string): Record<string, unknown> {
123 + if (!value || typeof value !== "object" || Array.isArray(value))
124 + fail(field, "must be a map");
125 + return value as Record<string, unknown>;
128 +function exactKeys(
129 + value: Record<string, unknown>,
130 + allowed: ReadonlySet<string>,
131 + field: string,
132 +): void {
133 + for (const key of Object.keys(value))
134 + if (!allowed.has(key))
135 + fail(field ? `${field}.${key}` : key, "unknown field");
138 +function requiredString(value: unknown, field: string): string {
139 + if (typeof value !== "string" || !value.trim())
140 + fail(field, "must be a non-empty string");
141 + return value;
144 +function capabilityList(value: unknown, field: string): readonly [] {
145 + if (value === undefined) return [];
146 + if (!Array.isArray(value)) fail(field, "must be a list of non-empty strings");
147 + value.forEach((item, index) => requiredString(item, `${field}[${index}]`));
148 + if (value.length > 0)
149 + fail(field, "must be empty because child capabilities start in slice 2");
150 + return [];
153 +function integer(
154 + value: unknown,
155 + field: string,
156 + fallback: number,
157 + minimum: number,
158 +): number {
159 + if (value === undefined) return fallback;
160 + if (!Number.isSafeInteger(value) || (value as number) < minimum)
161 + fail(field, `must be an integer >= ${minimum}`);
162 + return value as number;
165 +export function parseObserverDefinition(source: string): ObserverDefinition {
166 + const { yaml, prompt } = splitDocument(source);
167 + const document = parseDocument(yaml, {
168 + schema: "core",
169 + strict: true,
170 + uniqueKeys: true,
171 + });
172 + if (document.errors.length > 0)
173 + fail(
174 + "frontmatter",
175 + document.errors.map((error) => error.message).join("; "),
176 + );
177 + rejectAliases(document.contents as Node | null);
178 + if (!isMap(document.contents)) fail("frontmatter", "must be a YAML map");
179 + const value = document.toJS({ maxAliasCount: 0 }) as unknown;
180 + const root = record(value, "frontmatter");
181 + exactKeys(root, TOP_LEVEL, "");
183 + const id = requiredString(root.id, "id");
184 + if (!ID.test(id))
185 + fail(
186 + "id",
187 + "must be 1-64 lowercase letters, digits, or single hyphens, without edge hyphens",
188 + );
189 + const description =
190 + root.description === undefined
191 + ? undefined
192 + : requiredString(root.description, "description");
193 + try {
194 + return validateObserver(root, id, description, prompt);
195 + } catch (error) {
196 + if (error instanceof DefinitionError)
197 + throw new DefinitionError(error.field, error.message, id);
198 + throw error;
202 +function validateObserver(
203 + root: Record<string, unknown>,
204 + id: string,
205 + description: string | undefined,
206 + prompt: string,
207 +): ObserverDefinition {
208 + const trigger = record(root.trigger, "trigger");
209 + exactKeys(trigger, new Set(["manual"]), "trigger");
210 + if (trigger.manual !== true)
211 + fail("trigger.manual", "must be true in the currently supported trigger");
213 + const context = record(root.context, "context");
214 + exactKeys(context, new Set(["sources"]), "context");
215 + if (!Array.isArray(context.sources) || context.sources.length !== 0)
216 + fail("context.sources", "must be an empty list in the current slice");
218 + const agent = record(root.agent, "agent");
219 + exactKeys(
220 + agent,
221 + new Set([
222 + "model",
223 + "thinking",
224 + "tools",
225 + "skills",
226 + "prompts",
227 + "extensions",
228 + "limits",
229 + ]),
230 + "agent",
231 + );
232 + const model = requiredString(agent.model, "agent.model");
233 + if (!/^[^/\s]+\/[^/\s]+$/.test(model))
234 + fail("agent.model", "must use provider/model syntax");
235 + const thinking = requiredString(agent.thinking, "agent.thinking");
236 + if (!THINKING.has(thinking))
237 + fail(
238 + "agent.thinking",
239 + `unsupported thinking level ${JSON.stringify(thinking)}`,
240 + );
241 + const limitsInput =
242 + agent.limits === undefined ? {} : record(agent.limits, "agent.limits");
243 + exactKeys(limitsInput, new Set(Object.keys(DEFAULT_LIMITS)), "agent.limits");
244 + const limits = {
245 + timeoutMs: integer(
246 + limitsInput.timeoutMs,
247 + "agent.limits.timeoutMs",
248 + DEFAULT_LIMITS.timeoutMs,
249 + 1,
250 + ),
251 + outputTokens: integer(
252 + limitsInput.outputTokens,
253 + "agent.limits.outputTokens",
254 + DEFAULT_LIMITS.outputTokens,
255 + 1,
256 + ),
257 + retries: integer(
258 + limitsInput.retries,
259 + "agent.limits.retries",
260 + DEFAULT_LIMITS.retries,
261 + 0,
262 + ),
263 + toolCalls: integer(
264 + limitsInput.toolCalls,
265 + "agent.limits.toolCalls",
266 + DEFAULT_LIMITS.toolCalls,
267 + 0,
268 + ),
269 + };
271 + const result = record(root.result, "result");
272 + exactKeys(result, new Set(["type"]), "result");
273 + if (result.type !== "text")
274 + fail("result.type", "only text results are supported in the current slice");
275 + const delivery = record(root.delivery, "delivery");
276 + exactKeys(delivery, new Set(["type"]), "delivery");
277 + if (delivery.type !== "widget")
278 + fail(
279 + "delivery.type",
280 + "only widget delivery is supported in the current slice",
281 + );
283 + return {
284 + id,
285 + description,
286 + trigger: { manual: true },
287 + context: { sources: [] },
288 + agent: {
289 + model,
290 + thinking: thinking as ObserverDefinition["agent"]["thinking"],
291 + tools: capabilityList(agent.tools, "agent.tools"),
292 + skills: capabilityList(agent.skills, "agent.skills"),
293 + prompts: capabilityList(agent.prompts, "agent.prompts"),
294 + extensions: capabilityList(agent.extensions, "agent.extensions"),
295 + limits,
296 + },
297 + result: { type: "text" },
298 + delivery: { type: "widget" },
299 + prompt,
300 + };
303 +async function markdownFiles(root: string): Promise<string[]> {
304 + const files: string[] = [];
305 + async function visit(directory: string): Promise<void> {
306 + let entries: Dirent[];
307 + try {
308 + entries = await readdir(directory, { withFileTypes: true });
309 + } catch (error) {
310 + if ((error as NodeJS.ErrnoException).code === "ENOENT") return;
311 + throw error;
313 + for (const entry of entries.sort((a, b) => a.name.localeCompare(b.name))) {
314 + const path = join(directory, entry.name);
315 + if (entry.isDirectory()) await visit(path);
316 + else if (entry.isFile() && entry.name.toLowerCase().endsWith(".md"))
317 + files.push(path);
320 + await visit(root);
321 + return files;
324 +async function loadObserver(
325 + file: string,
326 + root: string,
327 + scope: ObserverScope,
328 +): Promise<ResolvedObserver> {
329 + let source: string;
330 + try {
331 + source = await readFile(file, "utf8");
332 + } catch (error) {
333 + const diagnostic = {
334 + file,
335 + field: "file",
336 + kind: "invalid" as const,
337 + message: error instanceof Error ? error.message : "could not read file",
338 + };
339 + return {
340 + id: relative(root, file),
341 + file,
342 + scope,
343 + state: "invalid",
344 + diagnostics: [diagnostic],
345 + };
347 + try {
348 + const definition = parseObserverDefinition(source);
349 + return {
350 + id: definition.id,
351 + file,
352 + scope,
353 + state: "ready",
354 + diagnostics: [],
355 + definition,
356 + };
357 + } catch (error) {
358 + const id =
359 + error instanceof DefinitionError && error.id
360 + ? error.id
361 + : relative(root, file);
362 + const diagnostic = {
363 + file,
364 + field: error instanceof DefinitionError ? error.field : "frontmatter",
365 + kind: "invalid" as const,
366 + message:
367 + error instanceof Error ? error.message : "invalid observer definition",
368 + id,
369 + };
370 + return {
371 + id,
372 + file,
373 + scope,
374 + state: "invalid",
375 + diagnostics: [diagnostic],
376 + };
380 +async function loadScope(
381 + root: string,
382 + scope: ObserverScope,
383 +): Promise<ResolvedObserver[]> {
384 + // Reads are independent and Promise.all preserves the sorted file order.
385 + // Parallel I/O avoids paying one filesystem round trip per observer.
386 + const loaded = await Promise.all(
387 + (await markdownFiles(root)).map((file) => loadObserver(file, root, scope)),
388 + );
389 + const byId = new Map<string, ResolvedObserver[]>();
390 + for (const observer of loaded) {
391 + const matches = byId.get(observer.id) ?? [];
392 + matches.push(observer);
393 + byId.set(observer.id, matches);
395 + for (const [id, matches] of byId) {
396 + if (matches.length < 2) continue;
397 + for (const observer of matches) {
398 + const diagnostic = {
399 + file: observer.file,
400 + field: "id",
401 + kind: "invalid" as const,
402 + message: `duplicate ${scope} observer id ${JSON.stringify(id)}`,
403 + id,
404 + };
405 + observer.state = "invalid";
406 + observer.diagnostics = [...observer.diagnostics, diagnostic];
409 + return loaded;
412 +function resourceDiagnostics(
413 + observer: ResolvedObserver,
414 + ctx: ExtensionContext,
415 +): WitnessDiagnostic[] {
416 + const definition = observer.definition;
417 + if (!definition) return [];
418 + const [provider, model] = definition.agent.model.split("/", 2) as [
419 + string,
420 + string,
421 + ];
422 + if (ctx.modelRegistry.find(provider, model)) return [];
423 + return [
425 + file: observer.file,
426 + field: "agent.model",
427 + kind: "unavailable",
428 + message: `model ${JSON.stringify(definition.agent.model)} is unavailable`,
429 + id: observer.id,
430 + },
431 + ];
434 +export async function discoverObservers(options: {
435 + userRoot: string;
436 + projectRoot: string;
437 + ctx: ExtensionContext;
438 +}): Promise<DiscoveryResult> {
439 + const projectTrusted = options.ctx.isProjectTrusted();
440 + const [user, project] = await Promise.all([
441 + loadScope(options.userRoot, "user"),
442 + projectTrusted
443 + ? loadScope(options.projectRoot, "project")
444 + : Promise.resolve([]),
445 + ]);
446 + const definitionDiagnostics = [...user, ...project].flatMap(
447 + (observer) => observer.diagnostics,
448 + );
449 + const resolved = new Map(user.map((observer) => [observer.id, observer]));
450 + for (const projectObserver of project) {
451 + const userObserver = resolved.get(projectObserver.id);
452 + resolved.set(
453 + projectObserver.id,
454 + userObserver
455 + ? { ...projectObserver, replaces: userObserver.file }
456 + : projectObserver,
457 + );
459 + const observers = [...resolved.values()]
460 + .map((observer): ResolvedObserver => {
461 + if (observer.state !== "ready") return observer;
462 + const diagnostics = resourceDiagnostics(observer, options.ctx);
463 + return diagnostics.length
464 + ? { ...observer, state: "unavailable", diagnostics }
465 + : observer;
466 + })
467 + .sort((a, b) => a.id.localeCompare(b.id));
468 + const unavailableDiagnostics = observers.flatMap((observer) =>
469 + observer.state === "unavailable" ? observer.diagnostics : [],
470 + );
471 + return {
472 + observers,
473 + diagnostics: [...definitionDiagnostics, ...unavailableDiagnostics],
474 + projectTrusted,
475 + };

extensions/witness/index.ts

Added
new file mode 100644
1 +import { join } from "node:path";
2 +import {
3 + CONFIG_DIR_NAME,
4 + type ExtensionAPI,
5 + type ExtensionCommandContext,
6 + type ExtensionContext,
7 + getAgentDir,
8 +} from "@earendil-works/pi-coding-agent";
9 +import type { AutocompleteItem } from "@earendil-works/pi-tui";
10 +import {
11 + type DiscoveryResult,
12 + discoverObservers,
13 + type ResolvedObserver,
14 +} from "./definitions.js";
16 +const SUBCOMMANDS = ["list", "show", "validate"] as const;
17 +const EMPTY: DiscoveryResult = {
18 + observers: [],
19 + diagnostics: [],
20 + projectTrusted: false,
21 +};
23 +function sanitize(text: string): string {
24 + return text.replace(/[\u0000-\u0009\u000b-\u001f\u007f-\u009f]/g, "�");
27 +function observerRoots(ctx: ExtensionContext) {
28 + return {
29 + userRoot: join(getAgentDir(), "witness", "observers"),
30 + projectRoot: join(ctx.cwd, CONFIG_DIR_NAME, "witness"),
31 + ctx,
32 + };
35 +function formatDiagnostic(
36 + diagnostic: DiscoveryResult["diagnostics"][number],
37 +): string {
38 + return `${diagnostic.file}: ${diagnostic.field}: ${diagnostic.kind}: ${diagnostic.message}`;
41 +function formatObserver(observer: ResolvedObserver): string {
42 + const replacement = observer.replaces
43 + ? `, replaces ${observer.replaces}`
44 + : "";
45 + return `${observer.id}: ${observer.state} (${observer.scope}, ${observer.file}${replacement})`;
48 +export default function witnessExtension(pi: ExtensionAPI): void {
49 + let generation = 0;
50 + let result = EMPTY;
51 + let pending: Promise<DiscoveryResult> | undefined;
53 + const startDiscovery = (
54 + ctx: ExtensionContext,
55 + warn: boolean,
56 + ): Promise<DiscoveryResult> => {
57 + const currentGeneration = ++generation;
58 + const discovery = discoverObservers(observerRoots(ctx)).then((next) => {
59 + if (currentGeneration !== generation) return result;
60 + result = next;
61 + if (warn && next.diagnostics.length > 0) {
62 + ctx.ui.notify(
63 + `Witness disabled ${next.observers.filter((observer) => observer.state !== "ready").length} observer(s) after ${next.diagnostics.length} validation diagnostic(s). Run /witness validate for details.`,
64 + "warning",
65 + );
66 + }
67 + return next;
68 + });
69 + pending = discovery;
70 + const clearPending = () => {
71 + if (pending === discovery) pending = undefined;
72 + };
73 + void discovery.then(clearPending, clearPending);
74 + return discovery;
75 + };
77 + const currentResult = async (
78 + ctx: ExtensionContext,
79 + ): Promise<DiscoveryResult> => pending ?? startDiscovery(ctx, false);
81 + const notify = (
82 + ctx: ExtensionCommandContext,
83 + text: string,
84 + level: "info" | "warning" | "error" = "info",
85 + ) => ctx.ui.notify(sanitize(text), level);
87 + pi.registerCommand("witness", {
88 + description: "Inspect and validate configured Witness observers",
89 + getArgumentCompletions(prefix): AutocompleteItem[] | null {
90 + const normalized = prefix.trimStart();
91 + const show = /^show\s+(\S*)$/i.exec(normalized);
92 + const items = show
93 + ? result.observers
94 + .filter((observer) => observer.id.startsWith(show[1] ?? ""))
95 + .map((observer) => ({
96 + value: `show ${observer.id}`,
97 + label: observer.id,
98 + description: observer.state,
99 + }))
100 + : normalized.includes(" ")
101 + ? []
102 + : SUBCOMMANDS.filter((command) =>
103 + command.startsWith(normalized.toLowerCase()),
104 + ).map((command) => ({ value: command, label: command }));
105 + return items.length ? items : null;
106 + },
107 + async handler(args, ctx) {
108 + const [command = "", id, ...extra] = args.trim().split(/\s+/);
109 + if (
110 + !SUBCOMMANDS.some((candidate) => candidate === command) ||
111 + extra.length > 0 ||
112 + (command === "show" && !id) ||
113 + (command !== "show" && id)
114 + ) {
115 + notify(ctx, "Usage: /witness list | show <id> | validate", "error");
116 + return;
118 + let current: DiscoveryResult;
119 + try {
120 + current =
121 + command === "validate"
122 + ? await startDiscovery(ctx, false)
123 + : await currentResult(ctx);
124 + } catch (error) {
125 + notify(
126 + ctx,
127 + `Witness definition discovery failed: ${error instanceof Error ? error.message : String(error)}`,
128 + "error",
129 + );
130 + return;
132 + if (command === "list") {
133 + notify(
134 + ctx,
135 + current.observers.length
136 + ? current.observers.map(formatObserver).join("\n")
137 + : "No Witness observers configured.",
138 + );
139 + return;
141 + if (command === "validate") {
142 + notify(
143 + ctx,
144 + current.diagnostics.length
145 + ? current.diagnostics.map(formatDiagnostic).join("\n")
146 + : `Valid: ${current.observers.length} observer(s). Project definitions: ${current.projectTrusted ? "trusted and inspected" : "not trusted, ignored"}.`,
147 + current.diagnostics.length ? "warning" : "info",
148 + );
149 + return;
151 + const observer = current.observers.find(
152 + (candidate) => candidate.id === id,
153 + );
154 + if (!observer) {
155 + notify(ctx, `Unknown Witness observer: ${id}`, "error");
156 + return;
158 + const lines = [formatObserver(observer)];
159 + if (observer.definition)
160 + lines.push(JSON.stringify(observer.definition, null, 2));
161 + if (observer.diagnostics.length)
162 + lines.push(...observer.diagnostics.map(formatDiagnostic));
163 + notify(
164 + ctx,
165 + lines.join("\n"),
166 + observer.state === "ready" ? "info" : "warning",
167 + );
168 + },
169 + });
171 + pi.on("session_start", (_event, ctx) => {
172 + const discovery = startDiscovery(ctx, true);
173 + const currentGeneration = generation;
174 + void discovery.catch((error) => {
175 + if (currentGeneration !== generation) return;
176 + ctx.ui.notify(
177 + `Witness definition discovery failed: ${sanitize(error instanceof Error ? error.message : String(error))}`,
178 + "warning",
179 + );
180 + });
181 + });
183 + pi.on("session_shutdown", () => {
184 + generation++;
185 + pending = undefined;
186 + result = EMPTY;
187 + });
190 +export { discoverObservers, parseObserverDefinition } from "./definitions.js";

extensions/witness/mise.toml

Added
new file mode 100644
1 +[tasks.bench]
2 +description = "Benchmark Witness definition parsing and discovery"
3 +depends = ["//:install"]
4 +run = "node --no-warnings --experimental-strip-types __bench__/definitions.bench.ts"
6 +[tasks.profile]
7 +description = "Profile Witness definition parsing and discovery"
8 +depends = ["//:install"]
9 +run = [
10 + '''node -e "require('node:fs').mkdirSync('../../.research/witness', { recursive: true })"''',
11 + "node --no-warnings --experimental-strip-types --cpu-prof --cpu-prof-dir ../../.research/witness --cpu-prof-name witness.cpuprofile __bench__/definitions.bench.ts",
14 +[tasks.fmt]
15 +description = "Format witness extension"
16 +depends = ["//:install"]
17 +run = [
18 + "biome check --write --config-path ../.. .",
19 + "rumdl fmt --config ../../.rumdl.toml .",
20 + "taplo format --config ../../.taplo.toml mise.toml",
23 +[tasks.lint]
24 +description = "Lint witness extension"
25 +depends = ["//:install"]
26 +run = [
27 + "biome ci --config-path ../.. .",
28 + "rumdl check --config ../../.rumdl.toml .",
29 + "taplo format --check --config ../../.taplo.toml mise.toml",
32 +[tasks.test]
33 +description = "Test witness extension"
34 +depends = ["//:install"]
35 +run = [
36 + { task = "//:test", args = ["extensions/witness"] },
39 +[tasks.check]
40 +description = "Check witness extension"
41 +depends = ["//:install"]
42 +run = [
43 + { task = "//:build", args = ["witness"] },

extensions/witness/package.json

Added
new file mode 100644
1 +{
2 + "name": "@bugabinga/pi-ext-witness",
3 + "private": true,
4 + "type": "module",
5 + "main": "index.ts",
6 + "dependencies": {
7 + "yaml": "^2.9.0"
8 + },
9 + "peerDependencies": {
10 + "@earendil-works/pi-coding-agent": "*",
11 + "@earendil-works/pi-tui": "*"
12 + },
13 + "devDependencies": {
14 + "@marcfargas/pi-test-harness": "^0.6.1",
15 + "vitest": "^4.1.9"
16 + },
17 + "license": "MIT",
18 + "description": "Run configured background observers for Pi.",
19 + "keywords": [
20 + "pi",
21 + "pi-extension",
22 + "observers"
23 + ]

package-lock.json

Modified
540 540 "typebox": "*"
543 + "extensions/witness": {
544 + "name": "@bugabinga/pi-ext-witness",
545 + "license": "MIT",
546 + "dependencies": {
547 + "yaml": "^2.9.0"
548 + },
549 + "devDependencies": {
550 + "@marcfargas/pi-test-harness": "^0.6.1",
551 + "vitest": "^4.1.9"
552 + },
553 + "peerDependencies": {
554 + "@earendil-works/pi-coding-agent": "*",
555 + "@earendil-works/pi-tui": "*"
557 + },
543 558 "node_modules/@antfu/install-pkg": {
544 559 "version": "1.1.0",
545 560 "resolved": "https://registry.npmjs.org/@antfu/install-pkg/-/install-pkg-1.1.0.tgz",
1557 1572 "resolved": "extensions/web",
1558 1573 "link": true
1575 + "node_modules/@bugabinga/pi-ext-witness": {
1576 + "resolved": "extensions/witness",
1577 + "link": true
1578 + },
1560 1579 "node_modules/@chevrotain/types": {
1561 1580 "version": "11.1.2",
1562 1581 "resolved": "https://registry.npmjs.org/@chevrotain/types/-/types-11.1.2.tgz",