repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

feat: measure startup and extension context impact

225fa8120e74431ddf763130099c0d9ad328ca21

Oliver Jan Krylow <oliver@bugabinga.net> ·

SSH signature · not checked

Browse filesCompare with HEAD

Changed files (6)

docs/context-startup.md

Added
new file mode 100644
1 +# Startup context overview
3 +Run `mise run //:context-startup`, optionally followed by `-- /path/to/project`.
4 +Requires installed Pi on PATH and valid existing credentials.
5 +Makes three live first-turn requests with `Reply only OK. Do not use tools.`.
7 +For a local-only split, run `mise run //:context-startup-sources`.
8 +It estimates `APPEND_SYSTEM.md`, `pi/agent` skill entries, extension/package skill entries, and shared catalog instructions/wrappers separately, using installed Pi's formatter and token heuristic.
9 +It uses the configured zero-message startup, including extension `resources_discover` hooks and the same read-only credential backend, but sends no measurement prompt.
10 +Skills are classified using Pi's loaded source metadata: package or `extension:` sources count as extensions, agent-directory skills count as `pi/agent`, and direct project-local skills are excluded from attribution without changing discovery.
11 +Unknown visible sources and skill diagnostics fail the estimate rather than silently dropping entries.
12 +The catalog includes names, descriptions, and locations, not skill bodies or prompt templates.
13 +Shared instructions are counted once; each group is rounded independently, so group sums may differ slightly from the combined total.
14 +These numbers measure the startup skill catalog, not arbitrary first-turn text injected by extensions.
15 +Run `mise run //:context-startup-skill-descriptions` to rank all visible `pi/agent` skills by description length and display the largest five descriptions for review.
16 +That ranking estimates raw description text only, excluding names, paths, XML escaping, and catalog markup; equal-length descriptions sort by skill name.
18 +## Measurement contract
20 +| Case | Config | Mode | User messages | Measurement |
21 +| --- | --- | --- | --- | --- |
22 +| baseline+1msg | Stock Pi | Headless | 1 | Provider input tokens and estimated component tokens |
23 +| headless+0msg | Your config | Headless | 0 | Estimated startup tokens only |
24 +| headless+1msg | Your config | Headless | 1 | Provider input tokens and estimated component tokens |
25 +| headless+1msg+no_pi_config | Your extensions/settings, no agent-dir text resources | Headless | 1 | Provider input tokens and estimated component tokens |
27 +The configured zero-message run resolves the model and thinking level, which are then pinned for every request.
28 +All cases use the same cwd and installed Pi version in separate processes.
29 +The baseline uses empty in-memory settings, an empty agent directory, stock tools, and no discovered extensions, skills, prompts, themes, or context files.
30 +It retains Pi's built-in extensions and stock system prompt.
31 +A selected model absent from stock Pi fails the baseline rather than importing user model config.
32 +Configured runs honor existing project trust; the script never grants trust.
33 +`no_pi_config` removes the agent directory's context file (`AGENTS.md` or `CLAUDE.md`), `SYSTEM.md`, `APPEND_SYSTEM.md`, and resources under its `skills/` and `prompts/` directories using Pi's resource-loader overrides.
34 +It keeps the same real agent directory, settings, models, extension discovery, extension-specific settings, and extension-injected content.
35 +Project context, trusted project resources, themes, and package-provided skills/prompts remain unchanged.
36 +Explicitly configured resources outside those agent-directory paths are retained.
37 +This is text-resource isolation, not an empty settings directory; nothing is renamed, copied, or disabled on disk.
38 +Unresolved project trust blocks the comparison rather than silently dropping project config.
40 +All cases use Pi's SDK startup services, ephemeral sessions, offline startup catalogs, and Pi's read-only credential backend.
41 +Credentials are consumed internally by Pi, never copied or printed; expired OAuth credentials cannot be refreshed by this script.
42 +This bypasses CLI onboarding, migrations, and update checks, not the headless extension lifecycle.
43 +Extensions may still perform their normal state/log writes and background requests.
45 +## Reading the output
47 +`input tok` is provider-reported input plus cache reads and cache writes, excluding output tokens.
48 +Cached input occupies context too.
49 +`~sys tok`, `~tools tok`, and `~msgs tok` use the installed Pi SDK's `estimateTokens()` heuristic.
50 +For text, Pi uses `Math.ceil(text.length / 4)`, based on JavaScript string length, not UTF-8 bytes.
51 +System text and serialized active tool definitions are each estimated as text.
52 +Tool definitions include names, descriptions, and parameter schemas, not implementation code.
53 +Messages use Pi's per-message estimator directly, including its image allowance and per-message rounding.
54 +Empty components count as zero.
55 +Zero-message estimates are captured after startup hooks finish.
56 +One-message estimates are captured at the agent's stream boundary after context transformations, before provider serialization.
57 +No raw prompts, provider payloads, credentials, or response transcripts are saved by the script.
59 +Configured headless minus baseline measures total config overhead, including changed tool definitions.
60 +`Pi text resources` is configured headless minus `no_pi_config`, using provider input tokens.
61 +It measures the effect of removing those resources with extensions/settings held fixed, not the residual extensions-only overhead: project context still remains.
62 +Extensions that react to missing resources can also contribute to this difference.
63 +Configured first-request component estimates minus configured zero-message estimates show first-turn growth, including the short user message.
64 +All displayed sizes are tokens; `~` distinguishes heuristic estimates from provider-reported counts.
65 +Estimates need not sum to provider input because tokenization, provider serialization, and payload-rewriting hooks can differ.
66 +A single run is an overview, not a stability benchmark or per-extension attribution.
68 +TUI measurement was removed after two live comparisons matched configured headless input at 24,579 tokens.
69 +Recheck manually if UI-dependent context behavior changes.
70 +Every worker exits after measurement; no terminal window is opened.
71 +Temporary measurement files contain counts and run options only and are removed after collection.
72 +Failures, extra agent requests, tool calls, startup errors, or mismatched model/version/cwd prevent misleading diffs.
74 +## Per-extension measurement
76 +Run `mise run //:context-extensions`, optionally followed by `-- /path/to/project`.
77 +A zero-message startup discovers the current file-backed extensions from Pi's runtime, including registered tools, event-handler names, and queued provider ownership.
78 +There is no extension-name allowlist: newly installed extensions automatically enter the next measurement.
79 +Registration signals prioritize likely context contributors but cannot prove an extension is harmless; arbitrary startup side effects, unknown hooks, and lazy registration remain possible.
80 +Every discovered file extension is therefore measured, including uncertain cases.
82 +The script makes N+2 live requests: a full-set control, one fresh process omitting each of N extensions, and a repeated full-set control.
83 +All request runs explicitly load the discovered paths in the same order, minus the one omitted path.
84 +Omission happens before module evaluation, not by filtering registrations after an extension already ran.
85 +Pi's inline built-ins and the measurement guard remain loaded.
86 +Settings, project trust, model, thinking level, cwd, prompt, and static package resources stay fixed.
87 +An extension's dynamically discovered resources disappear with its discovery hook, but package-declared skills/prompts remain: this measures an extension module, not uninstalling its whole package.
89 +Positive savings mean removal reduces provider input; negative savings mean removal increases input.
90 +Heuristic system/tool/message savings are reported separately.
91 +Deltas are not additive because extensions can interact.
92 +A failing omission, dependency break, changed model, or incorrect extension set is a failed measurement, never zero savings.
93 +If the final full-set control fails or differs in provider input or local component estimates, all savings are withheld.
94 +Matching endpoint controls detect some drift, not every transient change; do not edit configuration during measurement.
95 +Each worker retains the existing two-minute timeout, count-only output, read-only credentials, and no-tool-execution guard.
96 +Temporary inventory/options/results are removed after collection; raw prompts and provider payloads are never saved.
98 +## On-demand guidance checkpoint
100 +Current measurement: Pi 0.85.1, `openai-codex/gpt-6-astra`, high thinking, 39 file-backed extensions.
101 +Provider input is 8,934 tokens, 477 below the earlier 9,411-token checkpoint.
102 +The 41-request extension profile produced matching full-set controls at 8,934 tokens.
103 +The no-Pi-text case is 4,857 provider tokens.
105 +| Extension | Earlier removal saving | Current removal saving |
106 +| --- | ---: | ---: |
107 +| Ultra | 617 | 353 |
108 +| Web | 561 | 546 |
109 +| The System | 393 | 288 |
110 +| Terse | 240 | 240 |
111 +| Git-safe | 210 | 120 |
113 +The other 34 extensions had zero fixed first-turn impact for this probe.
114 +These are provider-token snapshot comparisons, not source-isolated before/after runs; removal savings remain non-additive.
115 +Current first-turn heuristic estimates are 7,596 system tokens, 2,250 tool tokens, and 8 message tokens, not a breakdown of the provider total.
117 +## Terse retirement checkpoint
119 +With Pi 0.85.1 and `openai-codex/gpt-6-astra` at high thinking, replacing Terse with five response-style sentences in the global `APPEND_SYSTEM.md` reduced configured first-request provider input from 8,072 to 7,911 tokens, a saving of 161 tokens.
120 +The stock baseline remained 1,071 tokens; the no-Pi-text case fell from 4,716 to 4,476 tokens.
121 +The configured system estimate fell from 6,847 to 6,713 heuristic tokens; tools and messages were unchanged.
122 +These before/after startup runs measure the combined migration, not response quality or an additive per-extension profile.
123 +The extension set now contains 38 packages; the earlier Terse profile above is historical.
125 +## Checks
127 +Run `mise run //:context-startup-test` for formatting, unit checks, and controlled subprocess fixtures.
128 +Run `mise run //:context-startup-fmt` to format only these measurement files.
129 +A real overview run additionally exercises installed Pi; fixture checks alone do not prove live behavior.
130 +Run `mise run //:context-guidance-check` to verify read-only live activation of tool-owned mockup guidance.
131 +It requires the configured model; it plans previews without writing files or opening a browser.
132 +Run `mise run //extensions/ultra:authoring-check` for the corresponding workflow authoring check.

scripts/context-extensions.mjs

Added
new file mode 100644
1 +import { mkdtemp, rm } from "node:fs/promises";
2 +import { homedir, tmpdir } from "node:os";
3 +import { join, resolve } from "node:path";
4 +import { pathToFileURL } from "node:url";
5 +import { comparable, runWorker } from "./context-startup.mjs";
7 +// Signals prioritize runs, never certify that an extension is context-free.
8 +export function registrationSignals(extension) {
9 + const signals = [];
10 + if (extension.tools.length) signals.push("tools");
11 + if (extension.providers.length) signals.push("provider");
12 + for (const [event, label] of [
13 + ["before_agent_start", "prompt"],
14 + ["context", "context"],
15 + ["before_provider_request", "payload"],
16 + ["resources_discover", "resources"],
17 + ["input", "input"],
18 + ["session_start", "startup"],
19 + ])
20 + if (extension.hooks.includes(event)) signals.push(label);
21 + return signals.length
22 + ? signals
23 + : [extension.hooks.length ? "other-hooks" : "unknown"];
26 +export async function measureExtensions(
27 + cwd = process.cwd(),
28 + worker,
29 + progress = () => {},
30 +) {
31 + cwd = resolve(cwd);
32 + const agentDir =
33 + process.env.PI_CODING_AGENT_DIR ?? join(homedir(), ".pi", "agent");
34 + const directory = await mkdtemp(join(tmpdir(), "pi-context-extensions-"));
35 + let index = 0;
36 + const run = (kind, selection, extensionPaths) =>
37 + runWorker(
38 + {
39 + kind,
40 + cwd,
41 + agentDir,
42 + sourceAgentDir: agentDir,
43 + selection,
44 + extensionPaths,
45 + inspectExtensions: true,
46 + resultPath: join(directory, `${index++}.result.json`),
47 + },
48 + worker,
49 + );
50 + try {
51 + progress("Discovering configured extensions and startup selection");
52 + const startup = await run("headless+0msg");
53 + if (startup.error) return { error: startup.error, inventory: [], runs: [] };
54 + const inventory = startup.extensions;
55 + if (
56 + !Array.isArray(inventory) ||
57 + inventory.some(
58 + (entry) =>
59 + typeof entry.path !== "string" ||
60 + !entry.path ||
61 + !Array.isArray(entry.tools) ||
62 + !Array.isArray(entry.hooks) ||
63 + !Array.isArray(entry.providers),
64 + ) ||
65 + new Set(inventory.map((entry) => entry.path)).size !== inventory.length
66 + )
67 + return {
68 + error: "Missing or invalid runtime extension inventory",
69 + inventory: [],
70 + runs: [],
71 + };
72 + const paths = inventory.map((entry) => entry.path);
73 + progress(
74 + `${inventory.length} file extensions discovered; measuring all, including uncertain registrations`,
75 + );
76 + progress(`Full-set control (1/${inventory.length + 2})`);
77 + const baseline = await run("headless+1msg", startup, paths);
78 + if (baseline.error)
79 + return { error: baseline.error, inventory, baseline, runs: [] };
80 + const ordered = [...inventory].sort((a, b) => {
81 + const uncertain = (entry) =>
82 + ["other-hooks", "unknown"].includes(registrationSignals(entry)[0]);
83 + return Number(uncertain(a)) - Number(uncertain(b));
84 + });
85 + const runs = [];
86 + for (const extension of ordered) {
87 + progress(
88 + `Without ${extension.name} (${runs.length + 2}/${inventory.length + 2}) [${registrationSignals(extension).join(", ")}]`,
89 + );
90 + const result = await run(
91 + "headless+1msg",
92 + startup,
93 + paths.filter((path) => path !== extension.path),
94 + );
95 + runs.push({ extension, result });
96 + }
97 + progress(
98 + `Repeated full-set control (${inventory.length + 2}/${inventory.length + 2})`,
99 + );
100 + const after = await run("headless+1msg", startup, paths);
101 + const stable =
102 + comparable(baseline, after) &&
103 + baseline.inputTokens === after.inputTokens &&
104 + JSON.stringify(baseline.local) === JSON.stringify(after.local);
105 + return {
106 + inventory,
107 + baseline,
108 + after,
109 + runs,
110 + stable,
111 + ...(!stable
112 + ? { error: "Full-set control failed or changed; savings withheld" }
113 + : {}),
114 + };
115 + } finally {
116 + await rm(directory, { recursive: true, force: true });
120 +export function formatExtensionOverview(report) {
121 + const { baseline, after, runs } = report;
122 + if (!baseline || baseline.error)
123 + return `Extension measurement failed: ${report.error}`;
124 + const n = (value) => value.toLocaleString("en-US");
125 + const signed = (value) => `${value >= 0 ? "+" : ""}${n(value)}`;
126 + const valid = (result) =>
127 + report.stable &&
128 + comparable(baseline, result) &&
129 + Number.isSafeInteger(result.inputTokens);
130 + const label = (extension) =>
131 + report.inventory.filter((entry) => entry.name === extension.name).length > 1
132 + ? extension.path
133 + : extension.name;
134 + const width = Math.max(
135 + 17,
136 + ...report.inventory.map((entry) => label(entry).length),
137 + );
138 + const row = (cells) =>
139 + cells
140 + .map((cell, i) =>
141 + i ? String(cell).padStart(11) : String(cell).padEnd(width),
143 + .join(" ");
144 + const lines = [
145 + `Pi ${baseline.version} · ${baseline.provider}/${baseline.model} · ${baseline.thinking}`,
146 + baseline.cwd,
147 + `Full set: ${n(baseline.inputTokens)} input tok; repeat: ${after?.error ? "FAIL" : n(after.inputTokens)}`,
148 + "",
149 + row([
150 + "extension removed",
151 + "saved tok",
152 + "~sys saved",
153 + "~tool saved",
154 + "~msg saved",
155 + ]),
156 + "─".repeat(width + 48),
157 + ];
158 + const ordered = [...runs].sort((a, b) => {
159 + if (valid(a.result) !== valid(b.result)) return valid(a.result) ? -1 : 1;
160 + return valid(a.result) ? a.result.inputTokens - b.result.inputTokens : 0;
161 + });
162 + for (const { extension, result } of ordered) {
163 + lines.push(
164 + row(
165 + valid(result)
166 + ? [
167 + label(extension),
168 + signed(baseline.inputTokens - result.inputTokens),
169 + ...["system", "tools", "messages"].map((key) =>
170 + signed(baseline.local[key] - result.local[key]),
171 + ),
173 + : [label(extension), result.error ? "FAIL" : "n/a", "", "", ""],
174 + ),
175 + );
177 + lines.push(
178 + "",
179 + "Positive = tokens saved by removal; negative = removal adds tokens.",
180 + "Input is provider-reported including cache; ~ = Pi estimates. Deltas are not additive.",
181 + "Same config/resources; only one extension module omitted before loading. Built-ins retained.",
182 + "All discovered file extensions measured, including unknown/future registrations. No name allowlist.",
183 + );
184 + if (report.error) lines.push(report.error);
185 + for (const { extension, result } of runs)
186 + if (result.error) lines.push(`${label(extension)}: ${result.error}`);
187 + return lines.join("\n");
190 +if (
191 + process.argv[1] &&
192 + import.meta.url === pathToFileURL(resolve(process.argv[1])).href
193 +) {
194 + try {
195 + const report = await measureExtensions(
196 + process.argv[2],
197 + undefined,
198 + (message) => console.error(message),
199 + );
200 + console.log(formatExtensionOverview(report));
201 + if (report.error || report.runs.some((run) => run.result.error))
202 + process.exitCode = 1;
203 + } catch {
204 + console.error(
205 + "Extension measurement failed: check installed Pi and measurement directory access.",
206 + );
207 + process.exitCode = 1;

scripts/context-extensions.test.ts

Added
new file mode 100644
1 +import {
2 + mkdirSync,
3 + mkdtempSync,
4 + readFileSync,
5 + rmSync,
6 + writeFileSync,
7 +} from "node:fs";
8 +import { tmpdir } from "node:os";
9 +import { join } from "node:path";
10 +import {
11 + DefaultResourceLoader,
12 + SettingsManager,
13 +} from "@earendil-works/pi-coding-agent";
14 +import { afterEach, expect, it, vi } from "vitest";
15 +import {
16 + formatExtensionOverview,
17 + measureExtensions,
18 + registrationSignals,
19 +} from "./context-extensions.mjs";
20 +import {
21 + inspectExtensionRegistrations,
22 + resourceOptions,
23 +} from "./context-startup-session.mjs";
25 +let directory: string | undefined;
26 +afterEach(() => {
27 + vi.unstubAllEnvs();
28 + if (directory) rmSync(directory, { recursive: true, force: true });
29 + directory = undefined;
30 +});
32 +it("inventories actual maps, provider ownership, and future registrations without configs or tool bodies", () => {
33 + const extension = (path: string) => ({
34 + path,
35 + tools: new Map(),
36 + handlers: new Map(),
37 + });
38 + const one = extension("/extensions/one/index.ts");
39 + one.tools.set("example", { description: "PRIVATE BODY" });
40 + one.handlers.set("future_event", [() => {}]);
41 + const result = {
42 + extensions: [
43 + one,
44 + extension("/extensions/new.ts"),
45 + extension("<inline:measurement>"),
46 + ],
47 + runtime: {
48 + pendingProviderRegistrations: [
49 + { name: "custom", extensionPath: one.path, config: "PRIVATE CONFIG" },
50 + ],
51 + pendingNativeProviderRegistrations: [
52 + {
53 + provider: { id: "native", other: "PRIVATE CONFIG" },
54 + extensionPath: one.path,
55 + },
56 + ],
57 + },
58 + };
59 + const inventory = inspectExtensionRegistrations(result);
60 + expect(inventory).toEqual([
61 + {
62 + path: one.path,
63 + name: "one",
64 + tools: ["example"],
65 + hooks: ["future_event"],
66 + providers: ["custom", "native"],
67 + },
68 + {
69 + path: "/extensions/new.ts",
70 + name: "new",
71 + tools: [],
72 + hooks: [],
73 + providers: [],
74 + },
75 + ]);
76 + expect(JSON.stringify(inventory)).not.toContain("PRIVATE");
77 + expect(
78 + inspectExtensionRegistrations({ ...result, runtime: {} }, inventory),
79 + ).toEqual(inventory);
80 + expect(registrationSignals(inventory[0])).toEqual(["tools", "provider"]);
81 + expect(registrationSignals(inventory[1])).toEqual(["unknown"]);
82 + expect(
83 + registrationSignals({ tools: [], providers: [], hooks: ["future_event"] }),
84 + ).toEqual(["other-hooks"]);
85 + expect(
86 + registrationSignals({
87 + tools: [],
88 + providers: [],
89 + hooks: [
90 + "before_agent_start",
91 + "context",
92 + "before_provider_request",
93 + "resources_discover",
94 + "input",
95 + "session_start",
96 + ],
97 + }),
98 + ).toEqual(["prompt", "context", "payload", "resources", "input", "startup"]);
99 +});
101 +it.each([true, false])(
102 + "omits modules before evaluation, retaining package skills and inline factories (keep file: %s)",
103 + async (keep) => {
104 + directory = mkdtempSync(join(tmpdir(), "pi-extension-profile-test-"));
105 + const cwd = join(directory, "project");
106 + const agentDir = join(directory, "agent");
107 + const packageDir = join(directory, "package");
108 + for (const path of [cwd, agentDir, packageDir]) mkdirSync(path);
109 + writeFileSync(
110 + join(packageDir, "package.json"),
111 + JSON.stringify({
112 + name: "fixture",
113 + pi: { extensions: ["omit.mjs"], skills: ["skill.md"] },
114 + }),
115 + );
116 + writeFileSync(
117 + join(packageDir, "omit.mjs"),
118 + "throw new Error('Excluded module executed'); export default () => {}; ",
119 + );
120 + writeFileSync(
121 + join(packageDir, "skill.md"),
122 + "---\nname: fixture\ndescription: Fixture skill\n---\nBody",
123 + );
124 + const kept = join(directory, "keep.mjs");
125 + writeFileSync(
126 + kept,
127 + "export default pi => pi.registerFlag('kept', { type: 'boolean', description: 'kept' });",
128 + );
129 + const settingsManager = SettingsManager.inMemory(
130 + { packages: [packageDir] },
131 + { projectTrusted: false },
132 + );
133 + const loader = new DefaultResourceLoader({
134 + cwd,
135 + agentDir,
136 + settingsManager,
137 + ...resourceOptions(
138 + "headless+1msg",
141 + name: "builtin",
142 + factory(pi) {
143 + pi.registerFlag("inline", {
144 + type: "boolean",
145 + description: "inline",
146 + });
147 + },
148 + },
149 + ],
150 + { extensionPaths: keep ? [kept] : [] },
151 + ),
152 + });
153 + await loader.reload();
154 + expect(loader.getExtensions().errors).toEqual([]);
155 + expect(
156 + loader.getExtensions().extensions.map((extension) => extension.path),
157 + ).toEqual([...(keep ? [kept] : []), "<inline:builtin>"]);
158 + expect(loader.getExtensions().extensions.at(-1)?.flags.has("inline")).toBe(
159 + true,
160 + );
161 + expect(loader.getSkills().skills.map((skill) => skill.name)).toContain(
162 + "fixture",
163 + );
164 + },
167 +function fixture() {
168 + directory = mkdtempSync(join(tmpdir(), "pi-extension-profile-test-"));
169 + const cwd = join(directory, "project with spaces");
170 + mkdirSync(cwd);
171 + const inventory = [
173 + path: "/extensions/writer/index.ts",
174 + name: "writer",
175 + tools: ["write"],
176 + hooks: [],
177 + providers: [],
178 + },
180 + path: "/extensions/observer/index.ts",
181 + name: "observer",
182 + tools: [],
183 + hooks: ["message_end"],
184 + providers: [],
185 + },
187 + path: "/extensions/brand-new/index.ts",
188 + name: "brand-new",
189 + tools: [],
190 + hooks: [],
191 + providers: [],
192 + },
194 + path: "/extensions/new-hook/index.ts",
195 + name: "new-hook",
196 + tools: [],
197 + hooks: ["future_event"],
198 + providers: [],
199 + },
200 + ];
201 + const log = join(directory, "calls.jsonl");
202 + const worker = join(directory, "worker.mjs");
203 + writeFileSync(
204 + worker,
206 + import fs from 'node:fs';
207 + import assert from 'node:assert/strict';
208 + const options = JSON.parse(fs.readFileSync(process.argv[2], 'utf8'));
209 + fs.appendFileSync(${JSON.stringify(log)}, JSON.stringify(options) + '\\n');
210 + assert.equal(process.cwd(), options.cwd);
211 + assert.equal(options.agentDir, options.sourceAgentDir);
212 + assert.equal(options.inspectExtensions, true);
213 + const inventory = ${JSON.stringify(inventory)};
214 + const mode = process.env.EXTENSION_PROFILE_TEST;
215 + const calls = fs.readFileSync(${JSON.stringify(log)}, 'utf8').trim().split('\\n').length;
216 + const omitted = inventory.find(entry => options.extensionPaths && !options.extensionPaths.includes(entry.path));
217 + const result = { kind: options.kind, cwd: options.cwd, version: 'test', provider: 'test', model: 'model', thinking: 'low', contextWindow: 100000,
218 + extensions: inventory.filter(entry => !options.extensionPaths || options.extensionPaths.includes(entry.path)),
219 + local: { system: 80, tools: omitted?.name === 'writer' ? 0 : 20, messages: 8, toolCount: 1 },
220 + ...(options.kind === 'headless+0msg' ? {} : { inputTokens: omitted?.name === 'writer' ? 80 : omitted?.name === 'brand-new' ? 105 : 100 }),
221 + };
222 + if (mode === 'startup' && options.kind === 'headless+0msg') result.error = 'controlled startup failure';
223 + if (mode === 'inventory') delete result.extensions;
224 + if (mode === 'omitted' && omitted?.name === 'writer') result.error = 'controlled dependency failure';
225 + if (mode === 'mismatch' && omitted?.name === 'writer') result.model = 'changed';
226 + if (mode === 'baseline' && calls === 2) result.error = 'controlled baseline failure';
227 + if (mode === 'drift' && calls === inventory.length + 3) result.inputTokens++;
228 + if (mode === 'repeat' && calls === inventory.length + 3) result.error = 'controlled repeat failure';
229 + fs.writeFileSync(options.resultPath, JSON.stringify(result));
230 + `,
231 + );
232 + return {
233 + cwd,
234 + worker,
235 + inventory,
236 + calls: () =>
237 + readFileSync(log, "utf8")
238 + .trim()
239 + .split("\n")
240 + .map((line) => JSON.parse(line)),
241 + };
244 +it("automatically measures every new extension, exactly one omission each, with pinned controls and cleanup", async () => {
245 + const { cwd, worker, inventory, calls } = fixture();
246 + const progress = vi.fn();
247 + const report = await measureExtensions(cwd, worker, progress);
248 + expect(report.stable).toBe(true);
249 + expect(report.inventory).toEqual(inventory);
250 + expect(report.runs.map((run) => run.extension.path).sort()).toEqual(
251 + inventory.map((entry) => entry.path).sort(),
252 + );
253 + const jobs = calls();
254 + expect(jobs).toHaveLength(inventory.length + 3);
255 + expect(jobs[0].extensionPaths).toBeUndefined();
256 + expect(jobs[1].extensionPaths).toEqual(inventory.map((entry) => entry.path));
257 + expect(jobs.at(-1).extensionPaths).toEqual(jobs[1].extensionPaths);
258 + for (const job of jobs.slice(1)) expect(job.selection.model).toBe("model");
259 + for (const [i, run] of report.runs.entries())
260 + expect(jobs[i + 2].extensionPaths).toEqual(
261 + inventory
262 + .filter((entry) => entry.path !== run.extension.path)
263 + .map((entry) => entry.path),
264 + );
265 + for (const job of jobs) expect(() => readFileSync(job.resultPath)).toThrow();
266 + expect(progress).toHaveBeenCalledTimes(inventory.length + 4);
267 + const text = formatExtensionOverview(report);
268 + expect(text).toMatch(/writer\s+\+20\s+\+0\s+\+20/);
269 + expect(text).toMatch(/brand-new\s+-5/);
270 + expect(text).toMatch(/observer\s+\+0/);
271 + expect(text).toContain("Deltas are not additive");
272 +});
274 +it.each(["startup", "inventory", "baseline"])(
275 + "blocks dependent requests after %s failure",
276 + async (mode) => {
277 + const { cwd, worker, calls } = fixture();
278 + vi.stubEnv("EXTENSION_PROFILE_TEST", mode);
279 + const report = await measureExtensions(cwd, worker);
280 + expect(report.error).toBeTruthy();
281 + expect(report.runs).toEqual([]);
282 + expect(calls()).toHaveLength(mode === "baseline" ? 2 : 1);
283 + expect(formatExtensionOverview(report)).toContain("failed");
284 + },
287 +it.each(["omitted", "mismatch"])(
288 + "reports %s failure without zero savings or blocking independent omissions",
289 + async (mode) => {
290 + const { cwd, worker } = fixture();
291 + vi.stubEnv("EXTENSION_PROFILE_TEST", mode);
292 + const report = await measureExtensions(cwd, worker);
293 + expect(report.stable).toBe(true);
294 + expect(report.runs).toHaveLength(4);
295 + expect(formatExtensionOverview(report)).toMatch(/writer\s+FAIL/);
296 + },
299 +it.each(["drift", "repeat"])(
300 + "withholds savings when final control has %s",
301 + async (mode) => {
302 + const { cwd, worker } = fixture();
303 + vi.stubEnv("EXTENSION_PROFILE_TEST", mode);
304 + const report = await measureExtensions(cwd, worker);
305 + expect(report.stable).toBe(false);
306 + expect(report.error).toContain("savings withheld");
307 + expect(formatExtensionOverview(report)).toMatch(/writer\s+n\/a/);
308 + },

scripts/context-startup-session.mjs

Added
new file mode 100644
1 +import { constants, existsSync } from "node:fs";
2 +import {
3 + access,
4 + readFile,
5 + realpath,
6 + rename,
7 + stat,
8 + writeFile,
9 +} from "node:fs/promises";
10 +import { homedir } from "node:os";
11 +import {
12 + basename,
13 + delimiter,
14 + dirname,
15 + extname,
16 + join,
17 + resolve,
18 + sep,
19 +} from "node:path";
20 +import { pathToFileURL } from "node:url";
21 +import { PROMPT, snapshot, summarize } from "./context-startup.mjs";
23 +// Resolve the installed executable, not this repository's SDK dependency.
24 +export async function installedPiDist() {
25 + for (const directory of (process.env.PATH ?? "").split(delimiter)) {
26 + const candidate = resolve(directory, "pi");
27 + try {
28 + await access(candidate, constants.X_OK);
29 + if (!(await stat(candidate)).isFile()) continue;
30 + } catch {
31 + continue;
32 + }
33 + let directoryPath = dirname(await realpath(candidate));
34 + while (true) {
35 + try {
36 + const manifest = JSON.parse(
37 + await readFile(join(directoryPath, "package.json"), "utf8"),
38 + );
39 + if (manifest.name === "@earendil-works/pi-coding-agent")
40 + return join(directoryPath, "dist");
41 + } catch (error) {
42 + if (error.code !== "ENOENT")
43 + throw new Error("Could not locate installed Pi");
44 + }
45 + const parent = dirname(directoryPath);
46 + if (parent === directoryPath) break;
47 + directoryPath = parent;
48 + }
49 + break;
50 + }
51 + throw new Error("Could not locate installed Pi");
54 +export async function estimateSourceTokens(
55 + agentDir,
56 + sdk,
57 + { skills, diagnostics },
58 +) {
59 + const append = await readFile(join(agentDir, "APPEND_SYSTEM.md"), "utf8");
60 + if (diagnostics.length)
61 + throw new Error(
62 + "Skill catalog has diagnostics; refusing a partial estimate",
63 + );
64 + const tokens = (content) =>
65 + sdk.estimateTokens({ role: "user", content, timestamp: 0 });
66 + const groups = { piAgent: [], extensions: [] };
67 + let ignoredProjectSkills = 0;
68 + for (const skill of skills.filter((skill) => !skill.disableModelInvocation)) {
69 + const source = skill.sourceInfo;
70 + if (
71 + source?.origin === "package" ||
72 + source?.source?.startsWith("extension:")
73 + ) {
74 + groups.extensions.push(skill);
75 + } else if (
76 + resolve(skill.filePath).startsWith(`${resolve(agentDir, "skills")}${sep}`)
77 + ) {
78 + groups.piAgent.push(skill);
79 + } else if (source?.scope === "project") {
80 + ignoredProjectSkills++;
81 + } else {
82 + throw new Error("Unattributed skill source; refusing a partial estimate");
83 + }
84 + }
85 + const parts = (items) => {
86 + const text = sdk.formatSkillsForPrompt(items);
87 + if (!text) return { shared: "", entries: "" };
88 + const start = text.indexOf(" <skill>");
89 + const end = text.lastIndexOf("</available_skills>");
90 + if (start < 0 || end <= start)
91 + throw new Error("Pi skill catalog format changed");
92 + return {
93 + shared: text.slice(0, start) + text.slice(end),
94 + entries: text.slice(start, end),
95 + };
96 + };
97 + const piAgent = parts(groups.piAgent);
98 + const extensions = parts(groups.extensions);
99 + const shared = piAgent.shared || extensions.shared;
100 + const appendix = tokens(append);
101 + const catalog = tokens(shared + piAgent.entries + extensions.entries);
102 + return {
103 + appendix,
104 + descriptions: groups.piAgent
105 + .map(({ name, description }) => ({
106 + name,
107 + description,
108 + tokens: tokens(description),
109 + }))
110 + .sort(
111 + (a, b) =>
112 + b.description.length - a.description.length ||
113 + a.name.localeCompare(b.name),
114 + ),
115 + piAgent: { tokens: tokens(piAgent.entries), count: groups.piAgent.length },
116 + extensions: {
117 + tokens: tokens(extensions.entries),
118 + count: groups.extensions.length,
119 + },
120 + shared: tokens(shared),
121 + ignoredProjectSkills,
122 + catalog,
123 + total: appendix + catalog,
124 + };
127 +export function inspectExtensionRegistrations(
128 + { extensions, runtime },
129 + previous = [],
130 +) {
131 + const providers = [
132 + ...(runtime.pendingProviderRegistrations ?? []).map(
133 + ({ name, extensionPath }) => ({ name, path: extensionPath }),
134 + ),
135 + ...(runtime.pendingNativeProviderRegistrations ?? []).map(
136 + ({ provider, extensionPath }) => ({
137 + name: provider.id,
138 + path: extensionPath,
139 + }),
140 + ),
141 + ];
142 + return extensions
143 + .filter((extension) => !extension.path.startsWith("<inline:"))
144 + .map((extension) => {
145 + const path = extension.path;
146 + const file = basename(path, extname(path));
147 + return {
148 + path,
149 + name: file === "index" ? basename(dirname(path)) : file,
150 + tools: [...extension.tools.keys()].sort(),
151 + hooks: [...extension.handlers.keys()].sort(),
152 + providers: [
153 + ...new Set([
154 + ...(previous.find((entry) => entry.path === path)?.providers ?? []),
155 + ...providers
156 + .filter((entry) => entry.path === path)
157 + .map((entry) => entry.name),
158 + ]),
159 + ].sort(),
160 + };
161 + });
164 +export function resourceOptions(
165 + kind,
166 + extensionFactories,
167 + { cwd, agentDir, projectTrusted, extensionPaths } = {},
168 +) {
169 + if (kind === "headless+1msg+no_pi_config") {
170 + // Keep the real settings and extension discovery. Remove only agent-dir text resources.
171 + const root = resolve(agentDir);
172 + const outside = (path, directory) =>
173 + !resolve(path).startsWith(`${join(root, directory)}${sep}`);
174 + const projectPrompt = (name) => {
175 + const path = join(cwd, ".pi", name);
176 + return projectTrusted && existsSync(path) ? path : "";
177 + };
178 + const append = projectPrompt("APPEND_SYSTEM.md");
179 + return {
180 + extensionFactories,
181 + agentsFilesOverride: (base) => ({
182 + ...base,
183 + agentsFiles: base.agentsFiles.filter(
184 + (file) => dirname(resolve(file.path)) !== root,
185 + ),
186 + }),
187 + skillsOverride: (base) => ({
188 + ...base,
189 + skills: base.skills.filter((skill) =>
190 + outside(skill.filePath, "skills"),
191 + ),
192 + }),
193 + promptsOverride: (base) => ({
194 + ...base,
195 + prompts: base.prompts.filter((prompt) =>
196 + outside(prompt.filePath, "prompts"),
197 + ),
198 + }),
199 + systemPrompt: projectPrompt("SYSTEM.md"),
200 + appendSystemPrompt: append ? [append] : [],
201 + };
203 + return kind === "baseline+1msg"
204 + ? {
205 + noExtensions: true,
206 + noSkills: true,
207 + noPromptTemplates: true,
208 + noThemes: true,
209 + noContextFiles: true,
210 + systemPrompt: "",
211 + appendSystemPrompt: [],
212 + extensionFactories,
214 + : {
215 + extensionFactories,
216 + ...(extensionPaths === undefined
217 + ? {}
218 + : { noExtensions: true, additionalExtensionPaths: extensionPaths }),
219 + };
222 +export function observeRequests(agent, observe) {
223 + const stream = agent.streamFunction;
224 + if (typeof stream !== "function")
225 + throw new Error("Pi stream API unavailable");
226 + agent.streamFunction = (model, context, options) => {
227 + observe(context);
228 + return stream(model, context, options);
229 + };
232 +// Only counts and public run metadata cross the process boundary.
233 +export async function publish(path, value) {
234 + await writeFile(`${path}.tmp`, JSON.stringify(value), { mode: 0o600 });
235 + await rename(`${path}.tmp`, path);
238 +export async function runSession(options) {
239 + const { kind, cwd, agentDir, sourceAgentDir, resultPath, selection } =
240 + options;
241 + const baseline = kind === "baseline+1msg";
242 + process.env.PI_CODING_AGENT_DIR = agentDir;
243 + process.env.PI_OFFLINE = "1";
244 + process.env.AI_AGENT = "pi";
245 + process.env.PI_CODING_AGENT = "true";
246 + let runtime;
247 + let inventory = [];
248 + let stopThemeWatcher;
249 + let stage = "installed Pi loading";
250 + let published = false;
251 + const finish = async (value) => {
252 + if (published) return;
253 + published = true;
254 + if (options.sourcesOnly) {
255 + if (value.error) {
256 + console.error(value.error);
257 + return;
259 + const result = value.sources;
260 + const n = (value) => value.toLocaleString("en-US");
261 + if (options.skillDescriptions) {
262 + const entries = result.descriptions;
263 + console.log(
265 + `pi/agent: ${entries.length} visible skills, largest descriptions first`,
266 + "~tok skill (raw description only, Pi chars/4)",
267 + ...entries.map(
268 + (entry) => `${String(entry.tokens).padStart(4)} ${entry.name}`,
269 + ),
270 + "",
271 + "Largest five descriptions:",
272 + ...entries
273 + .slice(0, 5)
274 + .map((entry) => `${entry.name}: ${entry.description}`),
275 + ].join("\n"),
276 + );
277 + return;
279 + console.log(
281 + `APPEND_SYSTEM.md: ~${n(result.appendix)} tok`,
282 + `pi/agent skills (${result.piAgent.count}): ~${n(result.piAgent.tokens)} tok`,
283 + `Extension/package skills (${result.extensions.count}): ~${n(result.extensions.tokens)} tok`,
284 + `Shared catalog instructions/wrapper: ~${n(result.shared)} tok`,
285 + `Total: ~${n(result.total)} tok (Pi chars/4; group rounding may differ)`,
286 + `Project-local skills excluded: ${result.ignoredProjectSkills}; skill bodies and prompt templates excluded.`,
287 + ].join("\n"),
288 + );
289 + } else {
290 + await publish(resultPath, value);
292 + };
293 + try {
294 + const dist = await installedPiDist();
295 + const load = (path) => import(pathToFileURL(join(dist, path)).href);
296 + const [
297 + sdk,
298 + { ReadOnlyAuthStorage },
299 + { InMemoryCodingAgentModelsStore },
300 + { builtInExtensions },
301 + trust,
302 + { initTheme, stopThemeWatcher: stopWatcher },
303 + { convertToLlm },
304 + ] = await Promise.all([
305 + load("index.js"),
306 + load("core/auth-storage.js"),
307 + load("core/models-store.js"),
308 + load("extensions/index.js"),
309 + load("core/trust-manager.js"),
310 + load("modes/interactive/theme/theme.js"),
311 + load("core/messages.js"),
312 + ]);
313 + stopThemeWatcher = stopWatcher;
314 + const { version } = JSON.parse(
315 + await readFile(join(dist, "../package.json"), "utf8"),
316 + );
317 + stage = "project trust resolution (use normal Pi to establish trust first)";
318 + let projectTrusted = false;
319 + if (!baseline) {
320 + projectTrusted =
321 + !trust.hasTrustRequiringProjectResources(cwd) ||
322 + new trust.ProjectTrustStore(agentDir).get(cwd);
323 + if (projectTrusted === null) {
324 + const policy = sdk.SettingsManager.create(cwd, agentDir, {
325 + projectTrusted: false,
326 + }).getDefaultProjectTrust();
327 + if (policy === "ask") throw new Error();
328 + projectTrusted = policy === "always";
331 + const settingsManager = baseline
332 + ? sdk.SettingsManager.inMemory({}, { projectTrusted: false })
333 + : sdk.SettingsManager.create(cwd, agentDir, { projectTrusted });
334 + stage = "read-only authentication/model initialization";
335 + const modelRuntime = await sdk.ModelRuntime.create({
336 + credentials: new ReadOnlyAuthStorage(join(sourceAgentDir, "auth.json")),
337 + modelsPath: baseline ? null : join(agentDir, "models.json"),
338 + modelsStore: new InMemoryCodingAgentModelsStore(),
339 + allowModelNetwork: false,
340 + });
341 + let contextMode;
342 + let hasUI;
343 + let toolCalls = 0;
344 + let extensionErrors = 0;
345 + const observer = {
346 + name: "context-measurement",
347 + hidden: true,
348 + factory(pi) {
349 + pi.on("session_start", (_event, ctx) => {
350 + contextMode = ctx.mode;
351 + hasUI = ctx.hasUI;
352 + });
353 + // Keep schemas intact, but fail closed if the reply-only request tries tools.
354 + pi.on("tool_call", () => {
355 + toolCalls++;
356 + return {
357 + block: true,
358 + terminate: true,
359 + reason: "Context measurement permits no tool execution",
360 + };
361 + });
362 + },
363 + };
364 + stage = "resource/session initialization";
365 + runtime = await sdk.createAgentSessionRuntime(
366 + async ({ sessionManager, sessionStartEvent }) => {
367 + const services = await sdk.createAgentSessionServices({
368 + cwd,
369 + agentDir,
370 + settingsManager,
371 + modelRuntime,
372 + resourceLoaderOptions: {
373 + ...resourceOptions(kind, [...builtInExtensions, observer], {
374 + cwd,
375 + agentDir,
376 + projectTrusted,
377 + extensionPaths: options.extensionPaths,
378 + }),
379 + ...(options.inspectExtensions
380 + ? {
381 + extensionsOverride(base) {
382 + inventory = inspectExtensionRegistrations(base);
383 + return base;
384 + },
386 + : {}),
387 + },
388 + });
389 + if (
390 + services.diagnostics.some((d) => d.type === "error") ||
391 + services.resourceLoader.getExtensions().errors.length ||
392 + settingsManager.drainErrors().length ||
393 + modelRuntime.getError()
395 + throw new Error();
396 + const model = selection
397 + ? modelRuntime.getModel(selection.provider, selection.model)
398 + : undefined;
399 + if (selection && !model) {
400 + stage = baseline
401 + ? "selected model unavailable in stock Pi (no user model config imported)"
402 + : "configured model selection";
403 + throw new Error();
405 + const created = await sdk.createAgentSessionFromServices({
406 + services,
407 + sessionManager,
408 + sessionStartEvent,
409 + model,
410 + thinkingLevel: selection?.thinking,
411 + });
412 + return { ...created, services, diagnostics: services.diagnostics };
413 + },
414 + { cwd, agentDir, sessionManager: sdk.SessionManager.inMemory(cwd) },
415 + );
416 + const { session } = runtime;
417 + stage = "startup model selection";
418 + if (!session.model) throw new Error();
419 + session.extensionRunner.onError(() => {
420 + extensionErrors++;
421 + });
422 + let requests = 0;
423 + let local;
424 + const messages = [];
425 + observeRequests(session.agent, (context) => {
426 + requests++;
427 + local = snapshot(context, sdk.estimateTokens);
428 + });
429 + session.subscribe((event) => {
430 + if (event.type === "message_end" && event.message.role === "assistant")
431 + messages.push(event.message);
432 + });
433 + const metadata = () => ({
434 + kind,
435 + cwd,
436 + version,
437 + provider: session.model?.provider,
438 + model: session.model?.id,
439 + thinking: session.thinkingLevel,
440 + contextWindow: session.model?.contextWindow,
441 + mode: contextMode,
442 + hasUI,
443 + projectTrusted,
444 + ...(options.inspectExtensions ? { extensions: inventory } : {}),
445 + });
446 + const result = () => {
447 + if (requests !== 1 || toolCalls || extensionErrors) {
448 + return {
449 + kind,
450 + error: `Invalid run: ${requests} requests, ${toolCalls} tool calls, ${extensionErrors} extension errors`,
451 + };
453 + try {
454 + const usage = summarize(
455 + messages
456 + .map((message) => JSON.stringify({ type: "message_end", message }))
457 + .join("\n"),
458 + );
459 + return { ...metadata(), local, ...usage, requests };
460 + } catch {
461 + return {
462 + kind,
463 + error:
464 + "No valid single response/usage; check provider access and read-only credential validity",
465 + };
467 + };
468 + initTheme(settingsManager.getTheme(), false);
469 + stage = "headless startup hooks";
470 + await session.bindExtensions({
471 + mode: "json",
472 + onError: () => {
473 + extensionErrors++;
474 + },
475 + });
476 + if (contextMode !== "json" || hasUI || extensionErrors) throw new Error();
477 + if (options.inspectExtensions) {
478 + stage = "runtime extension inventory";
479 + inventory = inspectExtensionRegistrations(
480 + runtime.services.resourceLoader.getExtensions(),
481 + inventory,
482 + );
483 + if (
484 + options.extensionPaths !== undefined &&
485 + JSON.stringify(inventory.map((entry) => entry.path)) !==
486 + JSON.stringify(options.extensionPaths)
487 + ) {
488 + stage = "explicit extension set/order validation";
489 + throw new Error();
492 + if (kind === "headless+0msg") {
493 + if (requests || messages.length) throw new Error();
494 + stage = "skill source attribution";
495 + const sources = options.sourcesOnly
496 + ? await estimateSourceTokens(
497 + agentDir,
498 + sdk,
499 + runtime.services.resourceLoader.getSkills(),
501 + : undefined;
502 + await finish({
503 + ...metadata(),
504 + sources,
505 + local: snapshot(
507 + ...session.agent.state,
508 + messages: convertToLlm(session.messages),
509 + },
510 + sdk.estimateTokens,
511 + ),
512 + requests,
513 + });
514 + } else {
515 + stage = "provider request (read-only credentials must already be valid)";
516 + await session.prompt(PROMPT);
517 + stage = "single-response/usage validation";
518 + await finish(result());
520 + } catch {
521 + await finish({ kind, error: `${stage} failed` });
522 + process.exitCode = 1;
523 + } finally {
524 + for (const cleanup of [
525 + () => runtime?.dispose(),
526 + () => stopThemeWatcher?.(),
527 + ]) {
528 + try {
529 + await cleanup();
530 + } catch {
531 + process.exitCode = 1;
537 +if (
538 + process.argv[1] &&
539 + import.meta.url === pathToFileURL(resolve(process.argv[1])).href
540 +) {
541 + if (["--sources", "--skill-descriptions"].includes(process.argv[2])) {
542 + const agentDir =
543 + process.env.PI_CODING_AGENT_DIR ?? join(homedir(), ".pi", "agent");
544 + await runSession({
545 + kind: "headless+0msg",
546 + cwd: process.cwd(),
547 + agentDir,
548 + sourceAgentDir: agentDir,
549 + sourcesOnly: true,
550 + skillDescriptions: process.argv[2] === "--skill-descriptions",
551 + });
552 + } else {
553 + await runSession(JSON.parse(await readFile(process.argv[2], "utf8")));
555 + process.exit(process.exitCode ?? 0);

scripts/context-startup.mjs

Added
new file mode 100644
1 +import { execFile } from "node:child_process";
2 +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
3 +import { homedir, tmpdir } from "node:os";
4 +import { dirname, join, resolve } from "node:path";
5 +import { fileURLToPath, pathToFileURL } from "node:url";
6 +import { promisify } from "node:util";
8 +const exec = promisify(execFile);
9 +export const PROMPT = "Reply only OK. Do not use tools.";
10 +export const CASES = [
11 + "baseline+1msg",
12 + "headless+0msg",
13 + "headless+1msg",
14 + "headless+1msg+no_pi_config",
15 +];
16 +const WORKER = join(
17 + dirname(fileURLToPath(import.meta.url)),
18 + "context-startup-session.mjs",
19 +);
21 +export function snapshot(
22 + { systemPrompt = "", tools = [], messages = [] },
23 + estimateTokens,
24 +) {
25 + const textTokens = (content) =>
26 + estimateTokens({ role: "user", content, timestamp: 0 });
27 + const definitions = tools.map(({ name, description, parameters }) => ({
28 + name,
29 + description,
30 + parameters,
31 + }));
32 + return {
33 + system: textTokens(systemPrompt),
34 + tools: definitions.length ? textTokens(JSON.stringify(definitions)) : 0,
35 + messages: messages.reduce(
36 + (total, message) => total + estimateTokens(message),
37 + 0,
38 + ),
39 + toolCount: definitions.length,
40 + };
43 +export function summarize(stdout) {
44 + const messages = [];
45 + for (const line of stdout.split("\n")) {
46 + let event;
47 + try {
48 + event = JSON.parse(line);
49 + } catch {
50 + continue;
51 + }
52 + if (event?.type === "message_end" && event.message?.role === "assistant")
53 + messages.push(event.message);
54 + }
55 + if (messages.length !== 1)
56 + throw new Error(`Expected one assistant response, got ${messages.length}`);
57 + const message = messages[0];
58 + if (message.stopReason !== "stop")
59 + throw new Error(
60 + "Pi did not complete a plain response; measurement rejected",
61 + );
62 + const { input, cacheRead, cacheWrite, output } = message.usage ?? {};
63 + if (
64 + ![input, cacheRead, cacheWrite, output].every(
65 + (value) => Number.isSafeInteger(value) && value >= 0,
66 + ) ||
67 + input + cacheRead + cacheWrite === 0
68 + )
69 + throw new Error("Missing or invalid provider token usage");
70 + return {
71 + provider: message.provider,
72 + model: message.model,
73 + inputTokens: input + cacheRead + cacheWrite,
74 + uncachedInputTokens: input,
75 + cacheReadTokens: cacheRead,
76 + cacheWriteTokens: cacheWrite,
77 + outputTokens: output,
78 + };
81 +export function comparable(a, b) {
82 + return (
83 + a &&
84 + b &&
85 + !a.error &&
86 + !b.error &&
87 + ["provider", "model", "thinking", "version", "cwd", "contextWindow"].every(
88 + (key) => a[key] !== undefined && a[key] === b[key],
89 + )
90 + );
93 +export function formatOverview(results) {
94 + const get = (kind) => results.find((row) => row.kind === kind);
95 + const base = get(CASES[0]);
96 + const zero = get(CASES[1]);
97 + const headless = get(CASES[2]);
98 + const noPiConfig = get(CASES[3]);
99 + const labelWidth = Math.max(...CASES.map((kind) => kind.length));
100 + const n = (value) => value.toLocaleString("en-US");
101 + const signed = (value) => `${value >= 0 ? "+" : ""}${n(value)}`;
102 + const row = (cells) =>
103 + cells
104 + .map((cell, i) =>
106 + ? String(cell).padStart([labelWidth, 11, 10, 10, 10][i])
107 + : String(cell).padEnd(labelWidth),
109 + .join(" ");
110 + const lines = [
111 + !zero || zero.error
112 + ? "Pi context overview (incomplete)"
113 + : `Pi ${zero.version} · ${zero.provider}/${zero.model} · ${zero.thinking} · ${n(zero.contextWindow)} ctx`,
114 + zero?.cwd ?? "",
115 + "",
116 + row(["case", "input tok", "~sys tok", "~tools tok", "~msgs tok"]),
117 + "─".repeat(labelWidth + 45),
118 + ...CASES.map((kind) => {
119 + const r = get(kind);
120 + return row(
121 + !r || r.error
122 + ? [kind, "FAIL", "", "", ""]
123 + : [
124 + kind,
125 + r.inputTokens === undefined ? "n/a" : n(r.inputTokens),
126 + n(r.local.system),
127 + n(r.local.tools),
128 + n(r.local.messages),
129 + ],
130 + );
131 + }),
132 + "",
133 + ];
134 + const delta = (label, a, b) => {
135 + if (
136 + !comparable(a, b) ||
137 + a.inputTokens === undefined ||
138 + b.inputTokens === undefined
140 + return `${label}: n/a (failed or mismatched runs)`;
141 + return `${label}: ${signed(a.inputTokens - b.inputTokens)} tok (${signed(Math.round(((a.inputTokens - b.inputTokens) / b.inputTokens) * 100))}%)`;
142 + };
143 + lines.push(delta("Config", headless, base));
144 + lines.push(delta("Pi text resources", headless, noPiConfig));
145 + lines.push(
146 + "no_pi_config keeps extension settings and project context; removes agent-dir instructions/skills/prompts.",
147 + );
148 + if (comparable(headless, zero)) {
149 + const changes = ["system", "tools", "messages"].map(
150 + (key) => `${key} ${signed(headless.local[key] - zero.local[key])}`,
151 + );
152 + lines.push(`First turn (~tok): ${changes.join(" · ")}`);
154 + lines.push(
155 + "Tokens: input = provider incl. cache; ~ = Pi heuristic (chars/4).",
156 + );
157 + lines.push("Estimates need not sum to provider input.");
158 + lines.push("SDK startup · offline catalogs · ephemeral · read-only auth");
159 + for (const r of results) if (r.error) lines.push(`${r.kind}: ${r.error}`);
160 + return lines.join("\n");
163 +export async function runWorker(options, worker = WORKER) {
164 + const configPath = `${options.resultPath}.options.json`;
165 + await writeFile(configPath, JSON.stringify(options), { mode: 0o600 });
166 + try {
167 + try {
168 + await exec(process.execPath, [worker, configPath], {
169 + cwd: options.cwd,
170 + timeout: 120_000,
171 + maxBuffer: 8 * 1024 * 1024,
172 + });
173 + } catch {
174 + /* Read the worker's sanitized failure, never echo subprocess logs. */
176 + const result = JSON.parse(await readFile(options.resultPath, "utf8"));
177 + if (
178 + options.selection &&
179 + !result.error &&
180 + !comparable(result, options.selection)
182 + return {
183 + kind: options.kind,
184 + error: "Model/thinking/version/cwd mismatch",
185 + };
186 + return result;
187 + } catch {
188 + return {
189 + kind: options.kind,
190 + error: "Process or measurement collection failed",
191 + };
195 +export async function measure(cwd = process.cwd(), worker = WORKER) {
196 + cwd = resolve(cwd);
197 + const sourceAgentDir =
198 + process.env.PI_CODING_AGENT_DIR ?? join(homedir(), ".pi", "agent");
199 + const directory = await mkdtemp(join(tmpdir(), "pi-context-overview-"));
200 + await mkdir(join(directory, "baseline"));
201 + const results = [];
202 + let selection;
203 + try {
204 + for (const kind of [CASES[1], CASES[0], CASES[2], CASES[3]]) {
205 + const resultPath = join(directory, `${kind}.result.json`);
206 + const options = {
207 + kind,
208 + cwd,
209 + sourceAgentDir,
210 + agentDir:
211 + kind === CASES[0] ? join(directory, "baseline") : sourceAgentDir,
212 + selection,
213 + resultPath,
214 + };
215 + const result = await runWorker(options, worker);
216 + results.push(result);
217 + if (kind === CASES[1]) {
218 + if (result.error) {
219 + for (const blocked of [CASES[0], CASES[2], CASES[3]])
220 + results.push({
221 + kind: blocked,
222 + error: "Blocked: startup model/config unresolved",
223 + });
224 + break;
226 + selection = result;
229 + return results;
230 + } finally {
231 + await rm(directory, { recursive: true, force: true });
235 +if (
236 + process.argv[1] &&
237 + import.meta.url === pathToFileURL(resolve(process.argv[1])).href
238 +) {
239 + try {
240 + const results = await measure(process.argv[2]);
241 + console.log(formatOverview(results));
242 + if (results.some((row) => row.error)) process.exitCode = 1;
243 + } catch {
244 + console.error(
245 + "Context overview failed: check installed Pi and measurement directory access.",
246 + );
247 + process.exitCode = 1;

scripts/context-startup.test.ts

Added
new file mode 100644
1 +import {
2 + chmodSync,
3 + mkdirSync,
4 + mkdtempSync,
5 + readdirSync,
6 + readFileSync,
7 + rmSync,
8 + writeFileSync,
9 +} from "node:fs";
10 +import { tmpdir } from "node:os";
11 +import { join } from "node:path";
12 +import { Agent } from "@earendil-works/pi-agent-core";
13 +import { createAssistantMessageEventStream } from "@earendil-works/pi-ai";
14 +import {
15 + DefaultResourceLoader,
16 + estimateTokens,
17 + formatSkillsForPrompt,
18 + loadSkills,
19 + SettingsManager,
20 +} from "@earendil-works/pi-coding-agent";
21 +import { afterEach, expect, it, vi } from "vitest";
22 +import {
23 + CASES,
24 + comparable,
25 + formatOverview,
26 + measure,
27 + PROMPT,
28 + snapshot,
29 + summarize,
30 +} from "./context-startup.mjs";
31 +import {
32 + estimateSourceTokens,
33 + installedPiDist,
34 + observeRequests,
35 + publish,
36 + resourceOptions,
37 +} from "./context-startup-session.mjs";
39 +const message = {
40 + role: "assistant",
41 + provider: "test",
42 + model: "test-model",
43 + stopReason: "stop",
44 + usage: { input: 100, cacheRead: 200, cacheWrite: 30, output: 2 },
45 +};
46 +const event = (value = message) =>
47 + JSON.stringify({ type: "message_end", message: value });
48 +const metadata = {
49 + cwd: "/project",
50 + version: "test",
51 + provider: "test",
52 + model: "test-model",
53 + thinking: "high",
54 + contextWindow: 100000,
55 +};
56 +let directory: string | undefined;
57 +afterEach(() => {
58 + vi.unstubAllEnvs();
59 + if (directory) rmSync(directory, { recursive: true, force: true });
60 + directory = undefined;
61 +});
63 +it("locates installed Pi from PATH without executing it", async () => {
64 + directory = mkdtempSync(join(tmpdir(), "pi-locator-test-"));
65 + const bin = join(directory, "bin");
66 + mkdirSync(bin);
67 + writeFileSync(
68 + join(directory, "package.json"),
69 + JSON.stringify({ name: "@earendil-works/pi-coding-agent" }),
70 + );
71 + const executable = join(bin, "pi");
72 + writeFileSync(executable, "This file must never execute.");
73 + chmodSync(executable, 0o755);
74 + vi.stubEnv("PATH", bin);
75 + expect(await installedPiDist()).toBe(join(directory, "dist"));
76 +});
77 +it("reports missing installed Pi without a repository SDK fallback", async () => {
78 + directory = mkdtempSync(join(tmpdir(), "pi-locator-test-"));
79 + vi.stubEnv("PATH", directory);
80 + await expect(installedPiDist()).rejects.toThrow(
81 + "Could not locate installed Pi",
82 + );
83 +});
85 +it("estimates global appendix and visible skill catalog, excluding bodies and rejecting incomplete catalogs", async () => {
86 + directory = mkdtempSync(join(tmpdir(), "pi-overview-test-"));
87 + const sdk = { estimateTokens, formatSkillsForPrompt };
88 + const measureSources = () =>
89 + estimateSourceTokens(
90 + directory,
91 + sdk,
92 + loadSkills({
93 + cwd: directory,
94 + agentDir: directory,
95 + skillPaths: [join(directory, "skills")],
96 + includeDefaults: false,
97 + }),
98 + );
99 + writeFileSync(join(directory, "APPEND_SYSTEM.md"), "漢漢漢漢");
100 + mkdirSync(join(directory, "skills"));
101 + expect(await measureSources()).toEqual({
102 + appendix: 1,
103 + descriptions: [],
104 + piAgent: { tokens: 0, count: 0 },
105 + extensions: { tokens: 0, count: 0 },
106 + shared: 0,
107 + ignoredProjectSkills: 0,
108 + catalog: 0,
109 + total: 1,
110 + });
111 + const visible = join(directory, "skills", "visible.md");
112 + const frontmatter = "---\nname: visible\ndescription: Read & write\n---\n";
113 + writeFileSync(visible, frontmatter + "Body");
114 + const result = await measureSources();
115 + const expectedCatalog = formatSkillsForPrompt([
117 + name: "visible",
118 + description: "Read & write",
119 + filePath: visible,
120 + baseDir: join(directory, "skills"),
121 + disableModelInvocation: false,
122 + sourceInfo: {
123 + path: visible,
124 + source: "user",
125 + scope: "user",
126 + origin: "top-level",
127 + },
128 + },
129 + ]);
130 + const expectedEntry = expectedCatalog.match(
131 + / {2}<skill>[\s\S]*? {2}<\/skill>\n/,
132 + )?.[0];
133 + if (!expectedEntry) throw new Error("Fixture catalog missing skill entry");
134 + expect(result).toEqual({
135 + appendix: 1,
136 + descriptions: [{ name: "visible", description: "Read & write", tokens: 3 }],
137 + piAgent: { tokens: Math.ceil(expectedEntry.length / 4), count: 1 },
138 + extensions: { tokens: 0, count: 0 },
139 + shared: Math.ceil((expectedCatalog.length - expectedEntry.length) / 4),
140 + ignoredProjectSkills: 0,
141 + catalog: Math.ceil(expectedCatalog.length / 4),
142 + total: 1 + Math.ceil(expectedCatalog.length / 4),
143 + });
144 + writeFileSync(visible, frontmatter + "Long skill body ".repeat(1000));
145 + writeFileSync(
146 + join(directory, "skills", "hidden.md"),
147 + "---\nname: hidden\ndescription: Not injected\ndisable-model-invocation: true\n---\nHidden body",
148 + );
149 + expect(await measureSources()).toEqual(result);
150 + writeFileSync(
151 + join(directory, "skills", "invalid.md"),
152 + "---\nname: INVALID\ndescription: Invalid uppercase name\n---\nBody",
153 + );
154 + await expect(measureSources()).rejects.toThrow(
155 + "Skill catalog has diagnostics",
156 + );
157 +});
159 +it("ranks only visible agent-dir descriptions by length, then name", async () => {
160 + directory = mkdtempSync(join(tmpdir(), "pi-overview-test-"));
161 + writeFileSync(join(directory, "APPEND_SYSTEM.md"), "");
162 + const skill = (
163 + name: string,
164 + description: string,
165 + sourceInfo = { source: "user", scope: "user", origin: "top-level" },
166 + ) => ({
167 + name,
168 + description,
169 + filePath: join(directory, "skills", `${name}.md`),
170 + disableModelInvocation: false,
171 + sourceInfo,
172 + });
173 + const result = await estimateSourceTokens(
174 + directory,
175 + { estimateTokens, formatSkillsForPrompt },
177 + diagnostics: [],
178 + skills: [
179 + skill("z", "漢漢漢漢"),
180 + skill("short", "abc"),
181 + skill("long", "abcde"),
182 + skill("a", "1234"),
183 + { ...skill("hidden", "x".repeat(100)), disableModelInvocation: true },
184 + skill("package", "x".repeat(100), {
185 + source: "fixture",
186 + scope: "user",
187 + origin: "package",
188 + }),
190 + ...skill("project", "x".repeat(100), {
191 + source: "project",
192 + scope: "project",
193 + origin: "top-level",
194 + }),
195 + filePath: join(directory, "project", "SKILL.md"),
196 + },
197 + ],
198 + },
199 + );
200 + expect(
201 + result.descriptions.map(({ name, tokens }) => ({ name, tokens })),
202 + ).toEqual([
203 + { name: "long", tokens: 2 },
204 + { name: "a", tokens: 1 },
205 + { name: "z", tokens: 1 },
206 + { name: "short", tokens: 1 },
207 + ]);
208 +});
210 +it("counts cached input exactly once and ignores notices, output, and other events", () => {
211 + expect(summarize(`notice\nnull\n{}\n${event()}\n`)).toEqual({
212 + provider: "test",
213 + model: "test-model",
214 + inputTokens: 330,
215 + uncachedInputTokens: 100,
216 + cacheReadTokens: 200,
217 + cacheWriteTokens: 30,
218 + outputTokens: 2,
219 + });
220 + expect(
221 + summarize(
222 + event({
223 + ...message,
224 + usage: { input: 100, cacheRead: 0, cacheWrite: 0, output: 2 },
225 + }),
226 + ).inputTokens,
227 + ).toBe(100);
228 +});
229 +it.each(["", `${event()}\n${event()}`])(
230 + "rejects missing or multiple responses",
231 + (stdout) => {
232 + expect(() => summarize(stdout)).toThrow("Expected one assistant response");
233 + },
235 +it.each(["error", "aborted", "toolUse", "length"])(
236 + "rejects %s responses",
237 + (stopReason) => {
238 + expect(() => summarize(event({ ...message, stopReason }))).toThrow(
239 + "measurement rejected",
240 + );
241 + },
243 +it.each([
244 + undefined,
245 + { input: 0, cacheRead: 0, cacheWrite: 0, output: 1 },
246 + { ...message.usage, input: -1 },
247 + { ...message.usage, cacheRead: "200" },
248 +])("rejects invalid usage", (usage) => {
249 + expect(() =>
250 + summarize(event({ ...message, usage } as typeof message)),
251 + ).toThrow("provider token usage");
252 +});
253 +it("uses Pi's character heuristic for system and tool definitions, excluding implementation and display fields", () => {
254 + const parameters = { type: "object" };
255 + expect(
256 + snapshot(
258 + systemPrompt: "漢漢漢漢",
259 + tools: [
261 + name: "read",
262 + description: "Read",
263 + parameters,
264 + label: "SECRET",
265 + execute() {},
266 + },
267 + ],
268 + messages: [],
269 + },
270 + estimateTokens,
271 + ),
272 + ).toEqual({
273 + system: 1,
274 + tools: Math.ceil(
275 + JSON.stringify([{ name: "read", description: "Read", parameters }])
276 + .length / 4,
277 + ),
278 + messages: 0,
279 + toolCount: 1,
280 + });
281 +});
282 +it("uses Pi's per-message rounding and image allowance, excluding message metadata", () => {
283 + expect(snapshot({}, estimateTokens)).toEqual({
284 + system: 0,
285 + tools: 0,
286 + messages: 0,
287 + toolCount: 0,
288 + });
289 + const messages = [
290 + { role: "user", content: "a", timestamp: 0 },
292 + role: "user",
293 + content: [
294 + { type: "text", text: "abcde" },
295 + { type: "image", data: "ignored", mimeType: "image/png" },
296 + ],
297 + timestamp: 0,
298 + },
299 + ];
300 + expect(snapshot({ messages }, estimateTokens).messages).toBe(1203);
301 + expect(snapshot({ systemPrompt: "abcde" }, estimateTokens).system).toBe(2);
302 +});
303 +it("baseline prevents every discovery route including SYSTEM and APPEND, configured preserves defaults", () => {
304 + const extensions = [];
305 + expect(resourceOptions("baseline+1msg", extensions)).toEqual({
306 + noExtensions: true,
307 + noSkills: true,
308 + noPromptTemplates: true,
309 + noThemes: true,
310 + noContextFiles: true,
311 + systemPrompt: "",
312 + appendSystemPrompt: [],
313 + extensionFactories: extensions,
314 + });
315 + expect(resourceOptions("headless+1msg", extensions)).toEqual({
316 + extensionFactories: extensions,
317 + });
318 +});
319 +it.each([
320 + { projectTrusted: true, projectPrompts: false },
321 + { projectTrusted: true, projectPrompts: true },
322 + { projectTrusted: false, projectPrompts: true },
323 +])(
324 + "removes only agent-dir text resources through Pi's real loader: %j",
325 + async ({ projectTrusted, projectPrompts }) => {
326 + directory = mkdtempSync(join(tmpdir(), "pi-overview-test-"));
327 + const cwd = join(directory, "project");
328 + const agentDir = join(directory, "agent");
329 + const packageDir = join(agentDir, "packages", "fixture");
330 + const put = (path: string, content: string) => {
331 + mkdirSync(join(path, ".."), { recursive: true });
332 + writeFileSync(path, content);
333 + };
334 + const skill = (name: string) =>
335 + `---\nname: ${name}\ndescription: Fixture skill\n---\nFixture body\n`;
336 + put(join(agentDir, "AGENTS.md"), "Global instructions");
337 + put(join(agentDir, "SYSTEM.md"), "Global system");
338 + put(join(agentDir, "APPEND_SYSTEM.md"), "Global append");
339 + put(join(agentDir, "skills", "global-fixture.md"), skill("global-fixture"));
340 + put(join(agentDir, "prompts", "global-fixture.md"), "Global template");
341 + put(join(cwd, "AGENTS.md"), "Project instructions");
342 + put(
343 + join(cwd, ".pi", "skills", "project-fixture.md"),
344 + skill("project-fixture"),
345 + );
346 + put(join(cwd, ".pi", "prompts", "project-fixture.md"), "Project template");
347 + if (projectPrompts) {
348 + put(join(cwd, ".pi", "SYSTEM.md"), "Project system");
349 + put(join(cwd, ".pi", "APPEND_SYSTEM.md"), "Project append");
351 + const settingsPath = join(agentDir, "settings.json");
352 + const settingsText = JSON.stringify({
353 + packages: [packageDir],
354 + contextFixture: "unchanged",
355 + });
356 + put(settingsPath, settingsText);
357 + put(
358 + join(packageDir, "package.json"),
359 + JSON.stringify({
360 + name: "context-fixture",
361 + pi: {
362 + extensions: ["index.mjs"],
363 + skills: ["skills"],
364 + prompts: ["prompts"],
365 + },
366 + }),
367 + );
368 + put(
369 + join(packageDir, "index.mjs"),
371 + import { readFileSync } from 'node:fs';
372 + export default pi => pi.registerFlag('fixture-setting', {
373 + type: 'boolean', description: JSON.parse(readFileSync(${JSON.stringify(settingsPath)}, 'utf8')).contextFixture
374 + });
375 + `,
376 + );
377 + put(
378 + join(packageDir, "skills", "extension-fixture.md"),
379 + skill("extension-fixture"),
380 + );
381 + put(
382 + join(packageDir, "prompts", "extension-fixture.md"),
383 + "Extension template",
384 + );
385 + const settingsManager = SettingsManager.create(cwd, agentDir, {
386 + projectTrusted,
387 + });
388 + const load = async (kind: string) => {
389 + const loader = new DefaultResourceLoader({
390 + cwd,
391 + agentDir,
392 + settingsManager,
393 + ...resourceOptions(kind, [], { cwd, agentDir, projectTrusted }),
394 + });
395 + await loader.reload();
396 + expect(loader.getExtensions().errors).toEqual([]);
397 + return loader;
398 + };
399 + const normal = await load("headless+1msg");
400 + const stripped = await load("headless+1msg+no_pi_config");
401 + const dynamicSkill = join(directory, "dynamic-fixture.md");
402 + put(dynamicSkill, skill("dynamic-fixture"));
403 + normal.extendResources({
404 + skillPaths: [
406 + path: dynamicSkill,
407 + metadata: {
408 + source: "extension:fixture",
409 + scope: "temporary",
410 + origin: "top-level",
411 + },
412 + },
413 + ],
414 + });
415 + const sources = await estimateSourceTokens(
416 + agentDir,
417 + { estimateTokens, formatSkillsForPrompt },
418 + normal.getSkills(),
419 + );
420 + expect(sources.piAgent.count).toBe(1);
421 + expect(sources.extensions.count).toBe(2);
422 + expect(sources.ignoredProjectSkills).toBe(projectTrusted ? 1 : 0);
423 + const included = normal
424 + .getSkills()
425 + .skills.filter((skill) => skill.sourceInfo.scope !== "project");
426 + expect(sources.catalog).toBe(
427 + Math.ceil(formatSkillsForPrompt(included).length / 4),
428 + );
429 + expect(sources.shared).toBeGreaterThan(0);
430 + expect(sources.extensions.tokens).toBeGreaterThan(0);
431 + const extensionOnly = await estimateSourceTokens(
432 + agentDir,
433 + { estimateTokens, formatSkillsForPrompt },
435 + skills: included.filter(
436 + (skill) =>
437 + skill.sourceInfo.origin === "package" ||
438 + skill.sourceInfo.source.startsWith("extension:"),
439 + ),
440 + diagnostics: [],
441 + },
442 + );
443 + expect(extensionOnly.shared).toBe(sources.shared);
444 + expect(extensionOnly.piAgent).toEqual({ tokens: 0, count: 0 });
445 + expect(extensionOnly.extensions).toEqual(sources.extensions);
446 + await expect(
447 + estimateSourceTokens(
448 + agentDir,
449 + { estimateTokens, formatSkillsForPrompt: () => "changed" },
450 + normal.getSkills(),
451 + ),
452 + ).rejects.toThrow("Pi skill catalog format changed");
453 + const unknown = {
454 + ...included[0],
455 + filePath: join(directory, "unknown.md"),
456 + sourceInfo: undefined,
457 + };
458 + await expect(
459 + estimateSourceTokens(
460 + agentDir,
461 + { estimateTokens, formatSkillsForPrompt },
462 + { skills: [unknown], diagnostics: [] },
463 + ),
464 + ).rejects.toThrow("Unattributed skill source");
465 + for (const loader of [normal, stripped]) {
466 + expect(
467 + loader
468 + .getExtensions()
469 + .extensions.find((ext) => ext.flags.has("fixture-setting"))
470 + ?.flags.get("fixture-setting")?.description,
471 + ).toBe("unchanged");
472 + expect(
473 + loader
474 + .getSkills()
475 + .skills.some((skill) => skill.name === "extension-fixture"),
476 + ).toBe(true);
477 + expect(
478 + loader
479 + .getPrompts()
480 + .prompts.some((prompt) => prompt.name === "extension-fixture"),
481 + ).toBe(true);
482 + expect(
483 + loader
484 + .getAgentsFiles()
485 + .agentsFiles.some((file) => file.path === join(cwd, "AGENTS.md")),
486 + ).toBe(true);
487 + expect(
488 + loader
489 + .getSkills()
490 + .skills.some((skill) => skill.name === "project-fixture"),
491 + ).toBe(projectTrusted);
492 + expect(
493 + loader
494 + .getPrompts()
495 + .prompts.some((prompt) => prompt.name === "project-fixture"),
496 + ).toBe(projectTrusted);
498 + expect(stripped.getExtensions().extensions.map((ext) => ext.path)).toEqual(
499 + normal.getExtensions().extensions.map((ext) => ext.path),
500 + );
501 + expect(
502 + normal
503 + .getSkills()
504 + .skills.some((skill) => skill.name === "global-fixture"),
505 + ).toBe(true);
506 + expect(
507 + normal
508 + .getPrompts()
509 + .prompts.some((prompt) => prompt.name === "global-fixture"),
510 + ).toBe(true);
511 + expect(
512 + normal
513 + .getAgentsFiles()
514 + .agentsFiles.some((file) => file.path === join(agentDir, "AGENTS.md")),
515 + ).toBe(true);
516 + for (let i = 0; i < 2; i++) {
517 + expect(
518 + stripped
519 + .getSkills()
520 + .skills.some((skill) => skill.name === "global-fixture"),
521 + ).toBe(false);
522 + expect(
523 + stripped
524 + .getPrompts()
525 + .prompts.some((prompt) => prompt.name === "global-fixture"),
526 + ).toBe(false);
527 + expect(
528 + stripped
529 + .getAgentsFiles()
530 + .agentsFiles.some(
531 + (file) => file.path === join(agentDir, "AGENTS.md"),
532 + ),
533 + ).toBe(false);
534 + expect(stripped.getSystemPrompt()).toBe(
535 + projectTrusted && projectPrompts ? "Project system" : undefined,
536 + );
537 + expect(stripped.getAppendSystemPrompt()).toEqual(
538 + projectTrusted && projectPrompts ? ["Project append"] : [],
539 + );
540 + if (i === 0) await stripped.reload();
542 + expect(readFileSync(settingsPath, "utf8")).toBe(settingsText);
543 + },
545 +it("shows only tokens with explicit heuristic labels and a first-turn estimate diff", () => {
546 + const rows = CASES.map((kind, i) => ({
547 + ...metadata,
548 + kind,
549 + local: {
550 + system: 1024 * (i + 1),
551 + tools: 2048,
552 + messages: i === 1 ? 2 : 1024,
553 + },
554 + ...(i === 1 ? {} : { inputTokens: [100, 0, 400, 250][i] }),
555 + }));
556 + const output = formatOverview(rows);
557 + expect(output).toContain("Config: +300 tok (+300%)");
558 + expect(output).toContain("Pi text resources: +150 tok (+60%)");
559 + expect(output).toContain("headless+1msg+no_pi_config");
560 + expect(output).not.toMatch(/TUI|tui|WezTerm|KiB|bytes/);
561 + expect(output).toContain("~sys tok ~tools tok");
562 + expect(output).toContain("~ = Pi heuristic (chars/4)");
563 + expect(output).toContain(
564 + "First turn (~tok): system +1,024 · tools +0 · messages +1,022",
565 + );
566 + expect(output).toContain("n/a");
567 + expect(
568 + new Set(
569 + output
570 + .split("\n")
571 + .slice(3, 9)
572 + .map((line) => line.length),
573 + ).size,
574 + ).toBe(1);
575 + rows[2].model = "different";
576 + expect(formatOverview(rows)).toContain(
577 + "Config: n/a (failed or mismatched runs)",
578 + );
579 + expect(comparable(rows[0], { ...rows[0], error: "failure" })).toBe(false);
580 +});
581 +it("publishes complete JSON atomically", async () => {
582 + directory = mkdtempSync(join(tmpdir(), "pi-overview-test-"));
583 + const result = join(directory, "result.json");
584 + await publish(result, { error: "startup failed" });
585 + expect(JSON.parse(readFileSync(result, "utf8"))).toEqual({
586 + error: "startup failed",
587 + });
588 + expect(readdirSync(directory)).toEqual(["result.json"]);
589 +});
591 +it("observes the real Agent stream boundary after context hooks without changing its response", async () => {
592 + const response = {
593 + ...message,
594 + role: "assistant" as const,
595 + api: "openai-responses" as const,
596 + stopReason: "stop" as const,
597 + content: [{ type: "text" as const, text: "OK" }],
598 + timestamp: 0,
599 + usage: {
600 + ...message.usage,
601 + totalTokens: 332,
602 + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
603 + },
604 + };
605 + const agent = new Agent({
606 + initialState: {
607 + model: {
608 + id: "test-model",
609 + name: "Test",
610 + provider: "test",
611 + api: "openai-responses",
612 + baseUrl: "https://example.invalid",
613 + reasoning: false,
614 + input: ["text"],
615 + contextWindow: 100000,
616 + maxTokens: 1000,
617 + cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
618 + },
619 + systemPrompt: "stock",
620 + },
621 + transformContext: async (messages) => [
622 + ...messages,
623 + { role: "user", content: "injected", timestamp: 0 },
624 + ],
625 + streamFn: () => {
626 + const stream = createAssistantMessageEventStream();
627 + stream.push({ type: "done", reason: "stop", message: response });
628 + stream.end();
629 + return stream;
630 + },
631 + });
632 + const observe = vi.fn();
633 + observeRequests(agent, observe);
634 + await agent.prompt(PROMPT);
635 + expect(observe).toHaveBeenCalledTimes(1);
636 + expect(observe.mock.calls[0][0].messages).toHaveLength(2);
637 + expect(agent.state.messages.at(-1)?.role).toBe("assistant");
638 + expect(() => observeRequests({}, observe)).toThrow(
639 + "Pi stream API unavailable",
640 + );
641 +});
643 +function fixtures() {
644 + directory = mkdtempSync(join(tmpdir(), "pi-overview-test-"));
645 + const cwd = join(directory, "project with spaces");
646 + mkdirSync(cwd);
647 + const shared = join(directory, "shared.cjs");
648 + writeFileSync(
649 + shared,
651 + const fs = require('node:fs');
652 + const path = require('node:path');
653 + const root = ${JSON.stringify(directory)};
654 + exports.log = value => fs.appendFileSync(path.join(root, 'calls.jsonl'), JSON.stringify(value) + '\\n');
655 + exports.result = options => ({ ...${JSON.stringify(metadata)}, cwd: options.cwd, kind: options.kind,
656 + mode: 'json', hasUI: false,
657 + local: { system: 1024, tools: 2048, messages: 2, toolCount: 4 },
658 + ...(options.kind === 'headless+0msg' ? {} : { inputTokens: options.kind === 'baseline+1msg' ? 100 : 400 }),
659 + });
660 + `,
661 + );
662 + const worker = join(directory, "worker.mjs");
663 + writeFileSync(
664 + worker,
666 + import fs from 'node:fs';
667 + import assert from 'node:assert/strict';
668 + import shared from ${JSON.stringify(shared)};
669 + const options = JSON.parse(fs.readFileSync(process.argv[2], 'utf8'));
670 + shared.log({ options });
671 + assert.equal(process.cwd(), options.cwd);
672 + if (options.kind === 'baseline+1msg') {
673 + assert.notEqual(options.agentDir, options.sourceAgentDir);
674 + assert.deepEqual(fs.readdirSync(options.agentDir), []);
676 + const fail = process.env.CONTEXT_TEST_FAIL === options.kind;
677 + const result = fail ? { kind: options.kind, error: 'controlled failure' } : shared.result(options);
678 + if (process.env.CONTEXT_TEST_FAIL === 'mismatch' && options.kind === 'baseline+1msg') result.model = 'other';
679 + fs.writeFileSync(options.resultPath, JSON.stringify(result));
680 + `,
681 + );
682 + // No terminal executable or control context is available to these subprocesses.
683 + vi.stubEnv("PATH", directory);
684 + vi.stubEnv("WEZTERM_PANE", "");
685 + const callsPath = join(directory, "calls.jsonl");
686 + return {
687 + cwd,
688 + worker,
689 + calls: () =>
690 + readFileSync(callsPath, "utf8")
691 + .trim()
692 + .split("\n")
693 + .map((line) => JSON.parse(line)),
694 + };
697 +it("runs four isolated headless cases without WezTerm, pins selection, and cleans measurement files", async () => {
698 + const { cwd, worker, calls } = fixtures();
699 + const result = await measure(cwd, worker);
700 + expect(result.map((row) => row.kind)).toEqual([
701 + CASES[1],
702 + CASES[0],
703 + CASES[2],
704 + CASES[3],
705 + ]);
706 + expect(result.some((row) => row.error)).toBe(false);
707 + expect(result.every((row) => row.mode === "json" && !row.hasUI)).toBe(true);
708 + const jobs = calls()
709 + .filter((call) => call.options)
710 + .map((call) => call.options);
711 + expect(
712 + jobs.slice(1).every((job) => job.selection.model === "test-model"),
713 + ).toBe(true);
714 + for (const job of jobs) expect(() => readFileSync(job.resultPath)).toThrow();
715 + expect(jobs).toHaveLength(4);
716 + expect(jobs[3].agentDir).toBe(jobs[3].sourceAgentDir);
717 +});
718 +it.each([
719 + "baseline+1msg",
720 + "headless+1msg",
721 + "headless+1msg+no_pi_config",
722 + "mismatch",
723 +])("reports %s failure without fabricated diffs", async (failure) => {
724 + const { cwd, worker } = fixtures();
725 + vi.stubEnv("CONTEXT_TEST_FAIL", failure);
726 + const result = await measure(cwd, worker);
727 + expect(result.some((row) => row.error)).toBe(true);
728 + expect(formatOverview(result)).toContain("n/a (failed or mismatched runs)");
729 +});
730 +it("blocks all paid runs when configured startup fails", async () => {
731 + const { cwd, worker, calls } = fixtures();
732 + vi.stubEnv("CONTEXT_TEST_FAIL", "headless+0msg");
733 + const result = await measure(cwd, worker);
734 + expect(result.every((row) => row.error)).toBe(true);
735 + expect(calls().filter((call) => call.options)).toHaveLength(1);
736 +});