repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

feat(ultra): add research and richer subagents

b0f9d61394a98479c06f603b2e6913854630bd7c

Oliver Krylow <o.krylow@isp-insoft.de> ·

SSH signature · not checked

Browse filesCompare with HEAD

Changed files (30)

extensions/ultra/README.md

Modified
8 8 ## Two surfaces
10 -- **`/ultra <name> [args]`** — interactive command. Runs a saved workflow behind a
11 - phase-first live overlay with keyboard phase toggling, agent focus, streaming
12 - transcripts, steering, targeted abort, and confirmed whole-run abort.
13 -- **`/ultra runs`** — browse, inspect, resume, stop, or delete saved runs.
14 -- **`/ultra <instructions>`** — injects the authoring prompt, enables the hidden
15 - `workflow` tool, and starts an agent turn to design or run a workflow.
16 -- **`workflow` tool** — model-callable after `/ultra` enables it. Pass a saved
10 +- **`/ultra`**: enables the hidden `workflow` tool for later agent turns, confirms activation, and does not start a turn.
11 +- **`/ultra exec <instructions>`**: enables the tool, injects the authoring prompt, and starts an agent turn to design or run a workflow.
12 +- **`/ultra run <name> [args]`**: runs a saved workflow behind a phase-first live overlay with keyboard phase toggling, agent focus, streaming transcripts, steering, targeted abort, and confirmed whole-run abort.
13 +- **`workflow` tool**: model-callable after `/ultra` enables it. Pass a saved
17 14 workflow `name` (+ optional `args`), or an inline `spec` object for a one-off
18 15 run. Returns the aggregated JSON; progress streams to a collapsible result
19 16 surface.
21 18 ## Workflows
23 -Use ultra for fan-out, multi-phase orchestration, or a single agent that specifically needs isolation, background execution, an alternate model, resumability, or strict structured output.
20 +Use ultra for fan-out, multi-phase orchestration, or a single agent that specifically needs isolation, an alternate model, resumability, or strict structured output.
24 21 For an ordinary one-agent task, perform the task directly instead of wrapping it in ultra.
26 23 A workflow is a single JSON object — phases (`single` / `fanout`), per-step
35 32 ### Authoring
37 -Run `/ultra <instructions>` to inject the authoring prompt and enable the
38 -`workflow` tool for the next agent turn.
34 +Run `/ultra exec <instructions>` to inject the authoring prompt and enable the `workflow` tool for the next agent turn.
39 35 It walks through co-designing and saving a workflow: phases, the read-only
40 36 fan-out rule, schemas + join-keys, interpolation, and where to save.
49 45 2. **global** — `~/.pi/agent/workflows/*.json`
50 46 3. **project** — `<cwd>/.pi/workflows/*.json`
52 -The shipped `review` workflow reviews the diff across dimensions, then
53 -adversarially verifies each finding — run it with `/ultra review [base-ref]`
54 -(no base ref reviews the working tree; a base ref reviews everything since it).
48 +The shipped `review` workflow reviews the diff across dimensions, adversarially verifies each finding, then synthesizes a concise analysis with concrete next steps.
49 +Run it with `/ultra run review [base-ref]` (no base ref reviews the working tree; a base ref reviews everything since it).
56 51 The shipped `feature` workflow drives a small feature end to end —
57 52 research → plan → adversarial critique → strengthened plan → implement → conditional security/quality review —
58 -run it with `/ultra feature <describe the task>` (the whole trailing text is the task).
53 +run it with `/ultra run feature <describe the task>` (the whole trailing text is the task).
59 54 A dedicated read-only phase rewrites the plan to resolve critique blockers before implementation.
60 55 Only its `implement` phase writes; every other phase is read-only.
62 57 The shipped `bug` workflow drives investigate → regression test → conditional fix → adversarial verification.
63 -Run it with `/ultra bug <describe the bug>`.
58 +Run it with `/ultra run bug <describe the bug>`.
64 59 The fix runs only after the regression phase proves the expected failure; verification runs only after a fix.
61 +The shipped `research` workflow drives question planning → parallel multi-source evidence gathering → gap and contradiction audit → targeted adversarial verification → cited synthesis.
62 +Run it with `/ultra run research <question>`.
63 +Its fan-out phases use the `web_search`, `web_fetch`, and `web_map` tools shipped by the `web` extension.
66 65 ## Results
68 -Completed review command and background runs show a human-readable summary, verified and refuted findings, drops, and recommended next steps. Expand the result to inspect raw JSON.
70 -The `workflow` tool accepts `background: true` to start a run and return immediately.
71 -When the detached run finishes, ultra sends its result back into the conversation.
72 -`/ultra runs` opens a saved-run picker for inspecting, resuming, stopping, or deleting runs.
73 -Run records live under `~/.pi/agent/ultra-runs/`.
67 +Completed review runs show synthesized analysis, verified and refuted findings, drops, and concrete recommended next steps.
68 +Expand the result only when raw JSON is needed.
75 70 Workflow runs append incomplete-run checkpoints to the current Pi session as custom entries that never enter LLM context.
76 71 If a run is aborted, the next matching run on the same session branch reuses completed steps and runs the missing or changed steps.
79 74 Both progress surfaces use the same phase-first hierarchy, authored step summaries, status language, icons, elapsed time, counts, checkpoint state, token usage, and cost.
80 75 The overlay also shows each exact resolved sub-agent prompt as the first user message in its transcript.
76 +Sub-agent tool calls keep one native Pi tool row from invocation through result and follow Pi's global expand state.
81 77 Collapsed workflow tool output remains counts-only with Pi's standard expand hint.
82 78 Expanded tool output summarizes every phase, automatically opens the active and failed phases, shows their agents, and ends with the human-readable workflow outcome.
83 79 The command overlay uses the same board, automatically collapses completed phases, and adds keyboard interaction.
102 98 `<cwd>/settings.json` (later overriding earlier), with the runtime settings
103 99 manager on top.
105 -| Key | Default | Meaning |
106 -| -------------------------- | ------- | -------------------------------------------------- |
107 -| `ultra.enabled` | `true` | When false, the tool + command refuse to run. |
108 -| `ultra.concurrency` | `5` | Max concurrent sub-agents in a `fanout` phase. |
109 -| `ultra.maxRetries` | `2` | Tool-less transport/output retries per step. |
110 -| `ultra.subagentExtensions` | `"all"` | Load host extensions into sub-agents, or not. |
111 -| `ultra.modelTiers` | `{}` | Optional `small` / `medium` / `big` model aliases. |
101 +| Key | Default | Meaning |
102 +| ----------------------------- | ------- | ---------------------------------------------------------- |
103 +| `ultra.enabled` | `true` | When false, the tool + command refuse to run. |
104 +| `ultra.concurrency` | `5` | Max concurrent sub-agents in a `fanout` phase. |
105 +| `ultra.maxRetries` | `2` | Tool-less transport/output retries per step. |
106 +| `ultra.subagentExtensions` | `"all"` | Load host extensions into sub-agents, or not. |
107 +| `ultra.subagentSkills` | `"all"` | Load host skills into sub-agents, or not. |
108 +| `ultra.modelTiers` | `{}` | Optional `small` / `medium` / `big` model aliases. |
109 +| `ultra.actionSummarizerModel` | `null` | Fast model for concise tool-action phrases; off when null. |
111 +`ultra.actionSummarizerModel` accepts a concrete `provider/model` ref.
112 +Ultra immediately renders a terse tool-aware action such as `Reading settings.ts`, then replaces it with the best-effort LLM phrase when available.
113 +Collapsed transcripts hide redundant tool previews; expand tools to inspect their complete native Pi arguments and results.
114 +Requests use bounded input, 40 output tokens, no retries, no prompt caching, and a two-second timeout.
115 +Rapid actions coalesce independently per agent, only one request runs at once, and an agent's final pending action flushes when it finishes.
116 +Workflow completion cancels leftover requests.
117 +Failures keep the deterministic fallback and never affect workflow execution.
118 +Unknown tool arguments and results use bounded top-level JSON summaries instead of raw JSON dumps.
119 +Tool arguments are sent to the configured model without secret redaction.
113 121 `ultra.modelTiers` maps workflow step `model` values of `small`, `medium`, or `big` to concrete model refs.
114 122 Unset tiers fall back to the session model.
118 126 ```json
120 128 "ultra": {
129 + "actionSummarizerModel": "openai/gpt-5-mini",
121 130 "modelTiers": {
122 131 "small": "openai/gpt-5-mini:low",
123 132 "medium": "anthropic/claude-sonnet-4:medium",
131 140 workflow sub-agents load the host's extensions. With `"all"`, a step can grant any
132 141 tool an installed extension provides (e.g. `web_search`); with `"none"`, sub-agents
133 142 are isolated to the core tools (`bash`, `find`, `grep`, `ls`, `read`, `ripgrep`).
134 -Either way the step's `tools` allowlist is the only gate on what's active, and
135 -ultra's own `workflow` tool is never grantable to a sub-agent. Workflows should
136 -name **tools**, never extensions — a workflow that lists a tool no installed
137 -extension provides simply runs without it.
143 +The step's `tools` allowlist gates extension tools, and ultra's own `workflow`
144 +tool is never grantable to a sub-agent. Workflows should name **tools**, never
145 +extensions. Ultra refuses to start a workflow when a listed tool is unavailable
146 +to its sub-agents.
148 +`ultra.subagentSkills` (`"all"` | `"none"`, default `"all"`) controls skill
149 +discovery for tool-using workflow sub-agents. Skills use Pi's normal global,
150 +project, package, and settings discovery, plus extension discovery when
151 +`ultra.subagentExtensions` is `"all"`. `"all"` also grants the
152 +built-in `read` tool so the agent can load selected `SKILL.md` files; every other
153 +tool remains workflow-gated. `"none"` omits all skills. Tool-less workflow steps
154 +are direct model calls rather than Pi agent sessions, so they cannot load skills.
139 156 ## Install / load

extensions/ultra/__tests__/action-summarizer.test.ts

Added
new file mode 100644
1 +import { describe, expect, it, vi } from "vitest";
2 +import { __test, createActionSummarizer } from "../action-summarizer.ts";
4 +function context() {
5 + const model = { provider: "fast", id: "tiny" };
6 + const find = vi.fn(() => model);
7 + const getApiKeyAndHeaders = vi.fn(async () => ({
8 + ok: true,
9 + apiKey: "key",
10 + headers: { "x-test": "yes" },
11 + }));
12 + return {
13 + model,
14 + find,
15 + getApiKeyAndHeaders,
16 + ctx: { modelRegistry: { find, getApiKeyAndHeaders } },
17 + };
20 +function fakeScheduler() {
21 + const timers: Array<() => void> = [];
22 + return {
23 + scheduler: {
24 + setTimeout(callback: () => void) {
25 + timers.push(callback);
26 + return callback;
27 + },
28 + clearTimeout(handle: () => void) {
29 + const index = timers.indexOf(handle);
30 + if (index >= 0) timers.splice(index, 1);
31 + },
32 + },
33 + flush() {
34 + for (const timer of timers.splice(0)) timer();
35 + },
36 + pending() {
37 + return timers.length;
38 + },
39 + };
42 +async function settle() {
43 + await new Promise((resolve) => setTimeout(resolve, 0));
46 +const input = (agentId: string, toolName = "read") => ({
47 + agentId,
48 + toolName,
49 + args: { path: "extensions/ultra/settings.ts" },
50 + taskSummary: "Inspect Ultra settings",
51 +});
53 +const response = (text: string) => ({
54 + stopReason: "stop",
55 + content: [{ type: "text", text }],
56 +});
58 +describe("action summarizer", () => {
59 + it("is disabled when the configured model is null", () => {
60 + const { ctx, find } = context();
61 + expect(createActionSummarizer(ctx as never, null)).toBeUndefined();
62 + expect(find).not.toHaveBeenCalled();
63 + });
65 + it("uses the configured cheap model with bounded low-latency options", async () => {
66 + const { ctx, model, find, getApiKeyAndHeaders } = context();
67 + const scheduler = fakeScheduler();
68 + const generate = vi.fn(async () => response('"Reading Ultra settings."\n'));
69 + const summarize = createActionSummarizer(ctx as never, "fast/tiny", {
70 + generate: generate as never,
71 + scheduler: scheduler.scheduler,
72 + });
73 + const pending = summarize?.({
74 + ...input("a"),
75 + args: { path: "extensions/ultra/settings.ts", noise: "x".repeat(3000) },
76 + });
78 + expect(generate).not.toHaveBeenCalled();
79 + scheduler.flush();
80 + await expect(pending).resolves.toBe("Reading Ultra settings");
81 + expect(find).toHaveBeenCalledWith("fast", "tiny");
82 + expect(getApiKeyAndHeaders).toHaveBeenCalledTimes(1);
83 + expect(generate).toHaveBeenCalledTimes(1);
84 + expect(generate.mock.calls[0][0]).toBe(model);
85 + expect(generate.mock.calls[0][1].systemPrompt).toContain(
86 + "Start with an -ing verb",
87 + );
88 + expect(
89 + generate.mock.calls[0][1].messages[0].content[0].text.length,
90 + ).toBeLessThan(1700);
91 + expect(generate.mock.calls[0][2]).toMatchObject({
92 + apiKey: "key",
93 + headers: { "x-test": "yes" },
94 + maxTokens: 40,
95 + maxRetries: 0,
96 + cacheRetention: "none",
97 + timeoutMs: 2000,
98 + });
99 + });
101 + it("coalesces rapid actions per agent to the latest call", async () => {
102 + const { ctx } = context();
103 + const scheduler = fakeScheduler();
104 + const generate = vi.fn(async () => response("Finding current references"));
105 + const summarize = createActionSummarizer(ctx as never, "fast/tiny", {
106 + generate: generate as never,
107 + scheduler: scheduler.scheduler,
108 + });
110 + const stale = summarize?.(input("a", "read"));
111 + const latest = summarize?.(input("a", "grep"));
112 + scheduler.flush();
114 + await expect(stale).resolves.toBeUndefined();
115 + await expect(latest).resolves.toBe("Finding current references");
116 + expect(generate).toHaveBeenCalledTimes(1);
117 + expect(generate.mock.calls[0][1].messages[0].content[0].text).toContain(
118 + "Tool: grep",
119 + );
120 + });
122 + it("debounces agents independently", () => {
123 + const { ctx } = context();
124 + const scheduler = fakeScheduler();
125 + const summarize = createActionSummarizer(ctx as never, "fast/tiny", {
126 + generate: vi.fn() as never,
127 + scheduler: scheduler.scheduler,
128 + });
130 + void summarize?.(input("a"));
131 + void summarize?.(input("b"));
132 + void summarize?.(input("a", "grep"));
134 + expect(scheduler.pending()).toBe(2);
135 + summarize?.dispose();
136 + });
138 + it("flushes a fast agent's latest action when it finishes", async () => {
139 + const { ctx } = context();
140 + const scheduler = fakeScheduler();
141 + const generate = vi.fn(async () => response("Reading Ultra settings"));
142 + const summarize = createActionSummarizer(ctx as never, "fast/tiny", {
143 + generate: generate as never,
144 + scheduler: scheduler.scheduler,
145 + });
146 + const pending = summarize?.(input("a"));
148 + summarize?.finish("a");
150 + await expect(pending).resolves.toBe("Reading Ultra settings");
151 + expect(scheduler.pending()).toBe(0);
152 + expect(generate).toHaveBeenCalledTimes(1);
153 + });
155 + it("allows only one model request in flight across agents", async () => {
156 + const { ctx } = context();
157 + const scheduler = fakeScheduler();
158 + const releases: Array<(value: unknown) => void> = [];
159 + const generate = vi.fn(
160 + () => new Promise((resolve) => releases.push(resolve)),
161 + );
162 + const summarize = createActionSummarizer(ctx as never, "fast/tiny", {
163 + generate: generate as never,
164 + scheduler: scheduler.scheduler,
165 + });
167 + const first = summarize?.(input("a"));
168 + const second = summarize?.(input("b"));
169 + scheduler.flush();
170 + await settle();
171 + expect(generate).toHaveBeenCalledTimes(1);
173 + releases.shift()?.(response("Reading first action"));
174 + await settle();
175 + expect(generate).toHaveBeenCalledTimes(2);
176 + releases.shift()?.(response("Reading second action"));
177 + await expect(first).resolves.toBe("Reading first action");
178 + await expect(second).resolves.toBe("Reading second action");
179 + });
181 + it("cancels queued work when its agent is aborted", async () => {
182 + const { ctx } = context();
183 + const scheduler = fakeScheduler();
184 + const generate = vi.fn(async () => response("Should not run"));
185 + const summarize = createActionSummarizer(ctx as never, "fast/tiny", {
186 + generate: generate as never,
187 + scheduler: scheduler.scheduler,
188 + });
189 + const pending = summarize?.(input("a"));
191 + summarize?.cancel?.("a");
192 + scheduler.flush();
193 + await expect(pending).resolves.toBeUndefined();
194 + expect(generate).not.toHaveBeenCalled();
195 + });
197 + it("cancels every pending action when disposed", async () => {
198 + const { ctx } = context();
199 + const scheduler = fakeScheduler();
200 + const generate = vi.fn(async () => response("Should not run"));
201 + const summarize = createActionSummarizer(ctx as never, "fast/tiny", {
202 + generate: generate as never,
203 + scheduler: scheduler.scheduler,
204 + });
205 + const a = summarize?.(input("a"));
206 + const b = summarize?.(input("b"));
208 + summarize?.dispose();
209 + scheduler.flush();
211 + await expect(a).resolves.toBeUndefined();
212 + await expect(b).resolves.toBeUndefined();
213 + expect(generate).not.toHaveBeenCalled();
214 + });
216 + it("fails silently when auth is unavailable", async () => {
217 + const { ctx } = context();
218 + ctx.modelRegistry.getApiKeyAndHeaders = vi.fn(async () => ({ ok: false }));
219 + const scheduler = fakeScheduler();
220 + const summarize = createActionSummarizer(ctx as never, "fast/tiny", {
221 + generate: vi.fn() as never,
222 + scheduler: scheduler.scheduler,
223 + });
224 + const pending = summarize?.(input("a"));
225 + scheduler.flush();
226 + await expect(pending).resolves.toBeUndefined();
227 + });
229 + it("rejects malformed model refs", () => {
230 + const { ctx } = context();
231 + expect(createActionSummarizer(ctx as never, "tiny")).toBeUndefined();
232 + expect(__test.parseModelRef("fast/tiny")).toEqual({
233 + provider: "fast",
234 + id: "tiny",
235 + });
236 + });
237 +});

extensions/ultra/__tests__/drivers.test.ts

Modified
1 +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
2 +import { tmpdir } from "node:os";
3 +import { join } from "node:path";
4 +import { DefaultResourceLoader } from "@earendil-works/pi-coding-agent";
1 5 import { describe, expect, it, vi } from "vitest";
2 6 import {
3 7 makeCompleteFn,
85 89 expect(session).toBe(fakeSession);
88 - it("loadExtensions:true flips noExtensions to false; other isolation flags stay true", async () => {
92 + it("loads extensions and skills independently while preserving other isolation", async () => {
89 93 const loaderOpts: Record<string, unknown>[] = [];
90 94 class FakeLoader {
91 95 reload = vi.fn().mockResolvedValue(undefined);
101 105 getAgentDir: () => "/agent",
102 106 cwd: "/work",
103 107 loadExtensions: true,
108 + loadSkills: true,
105 110 await factory({
106 111 model: {},
109 114 sessionManager: {},
112 - // only noExtensions flips; the other isolation flags are untouched.
113 117 expect(loaderOpts[0].noExtensions).toBe(false);
114 - expect(loaderOpts[0].noSkills).toBe(true);
118 + expect(loaderOpts[0].noSkills).toBe(false);
119 + expect(createAgentSession.mock.calls[0][0].tools).toEqual([
120 + "web_search",
121 + "structured_output",
122 + "read",
123 + ]);
115 124 expect(loaderOpts[0].noPromptTemplates).toBe(true);
116 125 expect(loaderOpts[0].noThemes).toBe(true);
117 126 expect(loaderOpts[0].noContextFiles).toBe(true);
130 +describe("sub-agent skill discovery", () => {
131 + it("loads skills by default and supports complete opt-out", async () => {
132 + const root = await mkdtemp(join(tmpdir(), "ultra-skills-"));
133 + const agentDir = join(root, "agent");
134 + const skillDir = join(agentDir, "skills", "ultra-skill-probe");
135 + await mkdir(skillDir, { recursive: true });
136 + await writeFile(
137 + join(skillDir, "SKILL.md"),
138 + "---\nname: ultra-skill-probe\ndescription: Verifies Ultra skill discovery.\n---\n\n# Probe\n",
139 + );
140 + try {
141 + const loader = new DefaultResourceLoader({
142 + cwd: root,
143 + agentDir,
144 + noExtensions: true,
145 + noSkills: false,
146 + noPromptTemplates: true,
147 + noThemes: true,
148 + noContextFiles: true,
149 + });
150 + await loader.reload();
151 + expect(loader.getSkills().skills.map((skill) => skill.name)).toContain(
152 + "ultra-skill-probe",
153 + );
155 + const disabled = new DefaultResourceLoader({
156 + cwd: root,
157 + agentDir,
158 + noExtensions: true,
159 + noSkills: true,
160 + noPromptTemplates: true,
161 + noThemes: true,
162 + noContextFiles: true,
163 + });
164 + await disabled.reload();
165 + expect(disabled.getSkills().skills).toEqual([]);
166 + } finally {
167 + await rm(root, { recursive: true, force: true });
169 + });
170 +});
121 172 describe("makeNewSessionManager", () => {
122 173 it("mints a fresh in-memory manager bound to cwd on every call", () => {
123 174 const made: unknown[] = [];

extensions/ultra/__tests__/engine.test.ts

Modified
92 92 // ---------------------------------------------------------------------------
94 94 describe("runWorkflow — phase dispatch", () => {
95 + it("disposes workflow-scoped runner resources", async () => {
96 + const fr = fakeRunner(() => ({ ok: true, value: true }));
97 + fr.runner.dispose = vi.fn();
98 + await runWorkflow(
99 + spec({
100 + phases: [
102 + id: "one",
103 + kind: "single",
104 + step: { summary: "Run", prompt: "Run", model: "m" },
105 + },
106 + ],
107 + }),
108 + undefined,
109 + baseDeps({ stepRunner: fr.runner }),
110 + );
112 + expect(fr.runner.dispose).toHaveBeenCalledTimes(1);
113 + });
95 115 it("single phase runs the step exactly once with item=undefined", async () => {
96 116 const fr = fakeRunner((args) => ({
97 117 ok: true,

extensions/ultra/__tests__/example.test.ts

Modified
24 24 // Pass the raw string so JSON.parse runs too — a malformed file fails loudly.
25 25 const spec = parseWorkflow(loadRaw());
26 26 expect(spec.name).toBe("review");
27 - expect(spec.phases).toHaveLength(2);
28 - expect(spec.phases[0].id).toBe("review");
29 - expect(spec.phases[1].id).toBe("verify");
27 + expect(spec.phases).toHaveLength(3);
28 + expect(spec.phases.map((phase) => phase.id)).toEqual([
29 + "review",
30 + "verify",
31 + "report",
32 + ]);
32 35 it("assigns compute deliberately by phase role and complexity", () => {
41 44 ).toEqual({
42 45 review: ["small", "medium"],
43 46 verify: ["big", "high"],
47 + report: ["big", "medium"],
47 51 it("declares a general review target positional and consumes {args.target}", () => {
48 52 const spec = parseWorkflow(loadRaw());
49 - // A single positional param soaks the whole tail: `/ultra review PR 123`,
50 - // `/ultra review src/foo.ts`, `/ultra review last 3 commits`, etc.
53 + // A single positional param soaks the whole tail: `/ultra run review PR 123`,
54 + // `/ultra run review src/foo.ts`, `/ultra run review last 3 commits`, etc.
51 55 expect(spec.args?.params).toEqual(["target"]);
52 56 const review = spec.phases.find((p) => p.id === "review");
53 57 expect(review?.step.prompt).toContain("{args.target}");
85 - it("the `return` selector correlates: it keeps ONLY real verdicts, each retaining its full join row", () => {
89 + it("returns the synthesized analysis, findings, and recommendations report", () => {
86 90 const spec = parseWorkflow(loadRaw());
87 - // Lockstep: drive the test with the file's actual `return` expression.
88 - expect(spec.return).toBe("{verify.results | where real}");
91 + expect(spec.return).toBe("{report.results}");
92 + const report = spec.phases.find((phase) => phase.id === "report");
93 + expect(report?.step.prompt).toContain("{review.results}");
94 + expect(report?.step.prompt).toContain("{verify.results}");
95 + expect(report?.step.prompt).toMatch(/analysis/i);
96 + expect(report?.step.prompt).toMatch(/recommendations/i);
97 + expect(report?.step.schema).toBe("ReviewReport");
90 - // Hand-built fanout-result fixture: some real, some not, each echoing `file`.
91 - const ctx = {
92 - results: {
93 - verify: [
94 - {
95 - title: "off-by-one in loop",
96 - real: true,
97 - why: "confirmed",
98 - file: "src/a.ts",
99 - },
100 - {
101 - title: "spurious nullcheck",
102 - real: false,
103 - why: "refuted",
104 - file: "src/b.ts",
105 - },
106 - {
107 - title: "unsanitized input",
108 - real: true,
109 - why: "reproduced",
110 - file: "src/c.ts",
111 - },
112 - ],
113 - },
99 + const value = {
100 + analysis: "The change is sound except for one confirmed defect.",
101 + findings: [
103 + title: "off-by-one in loop",
104 + file: "src/a.ts",
105 + why: "confirmed",
106 + severity: "high",
107 + },
108 + ],
109 + recommendations: ["Fix the loop bound and rerun focused tests."],
116 - const out = resolveSelector(spec.return ?? "", ctx) as Array<
117 - Record<string, unknown>
118 - >;
120 - // ONLY the real rows survive...
121 - expect(out).toHaveLength(2);
122 - // ...and each retains its WHOLE row (title+why+file together) so the verdict
123 - // still ties back to the finding it judged — not just a bare `file`.
124 - expect(out).toEqual([
125 - {
126 - title: "off-by-one in loop",
127 - real: true,
128 - why: "confirmed",
129 - file: "src/a.ts",
130 - },
131 - {
132 - title: "unsanitized input",
133 - real: true,
134 - why: "reproduced",
135 - file: "src/c.ts",
136 - },
137 - ]);
111 + expect(
112 + resolveSelector(spec.return ?? "", { results: { report: [value] } }),
113 + ).toEqual([value]);
140 116 it("is discoverable as a bundled workflow named `review`", () => {

extensions/ultra/__tests__/harness.test.ts

Modified
5 5 createTestSession,
6 6 type TestSession,
7 7 } from "@marcfargas/pi-test-harness";
8 -import { afterEach, describe, expect, it } from "vitest";
8 +import { afterEach, describe, expect, it, vi } from "vitest";
9 9 import { WorkflowSpecSchema } from "../spec.ts";
11 11 let t: TestSession | undefined;
33 33 expect(extension.commands.has("ultra-runs")).toBe(false);
36 - it("completes saved workflows and the runs subcommand from /ultra", async () => {
36 + it("completes subcommands first and saved workflows below run", async () => {
37 37 t = await createTestSession({ extensions: [ultraPath()] });
38 38 const [extension] = t.session.extensionRunner.extensions;
39 - const items = await extension.commands
40 - .get("ultra")
41 - .getArgumentCompletions("");
39 + const complete = extension.commands.get("ultra").getArgumentCompletions;
40 + const subcommands = await complete("");
41 + const workflows = await complete("run ");
43 - expect(items.map((item: { value: string }) => item.value)).toContain(
44 - "runs",
45 - );
46 - expect(items.map((item: { value: string }) => item.value)).toContain(
47 - "review",
43 + expect(subcommands.map((item: { value: string }) => item.value)).toEqual([
44 + "exec",
45 + "run",
46 + ]);
47 + expect(workflows.map((item: { value: string }) => item.value)).toEqual(
48 + expect.arrayContaining(["run research", "run review"]),
50 + });
52 + it("activates the workflow tool and notifies without starting a turn", async () => {
53 + t = await createTestSession({ extensions: [ultraPath()] });
54 + const [extension] = t.session.extensionRunner.extensions;
55 + const beforeMessages = t.session.messages.length;
56 + const beforeNotifications = t.events.uiCallsFor("notify").length;
58 + expect(t.session.getActiveToolNames()).not.toContain("workflow");
59 + await extension.commands
60 + .get("ultra")
61 + .handler("", t.session.extensionRunner.createCommandContext());
63 + expect(t.session.getActiveToolNames()).toContain("workflow");
64 + expect(t.session.messages).toHaveLength(beforeMessages);
65 + const notifications = t.events.uiCallsFor("notify");
66 + expect(notifications).toHaveLength(beforeNotifications + 1);
67 + expect(notifications.at(-1)?.args).toEqual([
68 + "ultra: workflow tool enabled.",
69 + "info",
70 + ]);
71 + });
73 + it("renders workflow progress as an overlay without replacing session content", async () => {
74 + t = await createTestSession({ extensions: [ultraPath()] });
75 + const [extension] = t.session.extensionRunner.extensions;
76 + const ctx = t.session.extensionRunner.createCommandContext();
77 + const custom = vi.fn().mockResolvedValue(null);
78 + Object.assign(ctx.ui, { custom });
80 + await extension.commands
81 + .get("ultra")
82 + .handler("run review inspect scrolling", ctx);
84 + expect(custom).toHaveBeenCalledOnce();
85 + expect(custom.mock.calls[0][1]).toEqual({
86 + overlay: true,
87 + overlayOptions: {
88 + anchor: "bottom-left",
89 + width: "100%",
90 + maxHeight: "100%",
91 + },
92 + });
93 + });
95 + it("refuses research when the web extension tools are unavailable", async () => {
96 + t = await createTestSession({ extensions: [ultraPath()] });
97 + const [extension] = t.session.extensionRunner.extensions;
98 + const ctx = t.session.extensionRunner.createCommandContext();
99 + const custom = vi.fn().mockResolvedValue(null);
100 + Object.assign(ctx.ui, { custom });
102 + await extension.commands.get("ultra").handler("run research question", ctx);
104 + expect(custom).not.toHaveBeenCalled();
105 + expect(t.events.uiCallsFor("notify").at(-1)?.args).toEqual([
106 + 'ultra: workflow "research" requires unavailable tools: web_fetch, web_map, web_search',
107 + "error",
108 + ]);
51 111 it("discovers a workflow saved after the extension loaded without /reload", async () => {
52 112 t = await createTestSession({ extensions: [ultraPath()] });
53 113 const [extension] = t.session.extensionRunner.extensions;
54 114 const complete = () =>
55 - extension.commands.get("ultra").getArgumentCompletions("");
115 + extension.commands.get("ultra").getArgumentCompletions("run ");
56 116 await complete();
58 118 const previousCwd = process.cwd();
73 133 const items = await complete();
74 134 expect(items.map((item: { value: string }) => item.value)).toContain(
75 - "fresh",
135 + "run fresh",
77 137 } finally {
78 138 process.chdir(previousCwd);
93 153 expect(tool.description).toContain(
94 154 "interpolation uses only the closed grammar",
156 + expect(tool.parameters.properties.background).toBeUndefined();
96 157 expect(stepSchema.required).toContain("summary");
97 158 expect(stepSchema.properties.summary.description).toContain("progress-UI");
98 159 expect(JSON.parse(JSON.stringify(specSchema))).toEqual(

extensions/ultra/__tests__/journal.test.ts

Modified
1 -import fs from "node:fs";
2 -import os from "node:os";
3 -import path from "node:path";
4 1 import { describe, expect, it, vi } from "vitest";
5 2 import {
6 3 createSessionWorkflowJournal,
7 - deleteWorkflowRun,
8 - listWorkflowRuns,
9 4 type SessionEntryLike,
10 - saveWorkflowRun,
11 5 } from "../journal.ts";
12 6 import type { WorkflowSpec } from "../spec.ts";
16 10 phases: [{ id: "p", kind: "single", step: { summary: "hi", prompt: "hi" } }],
19 -function tempHome(): string {
20 - return fs.mkdtempSync(path.join(os.tmpdir(), "ultra-journal-"));
23 -describe("saved workflow runs", () => {
24 - it("lists newest runs first and deletes by id", () => {
25 - const home = tempHome();
26 - saveWorkflowRun({
27 - home,
28 - id: "older",
29 - workflow: "review",
30 - cwd: "/work",
31 - startedAt: "2026-01-01T00:00:00.000Z",
32 - status: "completed",
33 - result: { findings: [] },
34 - });
35 - saveWorkflowRun({
36 - home,
37 - id: "newer",
38 - workflow: "feature",
39 - cwd: "/work",
40 - startedAt: "2026-01-02T00:00:00.000Z",
41 - status: "running",
42 - });
44 - expect(listWorkflowRuns(home).map((run) => run.id)).toEqual([
45 - "newer",
46 - "older",
47 - ]);
48 - expect(deleteWorkflowRun(home, "older")).toBe(true);
49 - expect(listWorkflowRuns(home).map((run) => run.id)).toEqual(["newer"]);
50 - });
52 - it("rejects unsafe run ids when deleting", () => {
53 - const home = tempHome();
54 - expect(deleteWorkflowRun(home, "../escape")).toBe(false);
55 - });
56 -});
58 13 describe("createSessionWorkflowJournal", () => {
59 14 function sessionStore() {
60 15 const entries: SessionEntryLike[] = [];

extensions/ultra/__tests__/progress.test.ts

Modified
103 + it("renders known and arbitrary JSON tool calls tersely", () => {
104 + let state = reduceProgress(
105 + initialReducerState(),
106 + map({
107 + type: "tool_execution_start",
108 + toolCallId: "read-1",
109 + toolName: "read",
110 + args: { path: "src/settings.ts", offset: 4, limit: 20 },
111 + }) as UltraProgressEvent,
112 + );
113 + expect(state.rows[0]?.action).toBe("Reading src/settings.ts");
115 + state = reduceProgress(
116 + state,
117 + map({
118 + type: "tool_execution_start",
119 + toolCallId: "custom-1",
120 + toolName: "custom_tool",
121 + args: {
122 + action: "sync",
123 + items: [1, 2, 3],
124 + options: { force: true },
125 + },
126 + }) as UltraProgressEvent,
127 + );
128 + expect(state.rows[0]?.action).toBe(
129 + "Using custom_tool · action=sync · items=[3] · options={1}",
130 + );
131 + });
133 + it("keeps complete tool results while rendering their JSON tersely", () => {
134 + const result = {
135 + content: [{ type: "text", text: "Verbose body that stays expandable" }],
136 + details: {
137 + ok: true,
138 + files: ["a.ts", "b.ts"],
139 + metadata: { source: "index" },
140 + },
141 + };
142 + let state = reduceProgress(
143 + initialReducerState(),
144 + map({
145 + type: "tool_execution_start",
146 + toolCallId: "custom-1",
147 + toolName: "custom_tool",
148 + args: { action: "index" },
149 + }) as UltraProgressEvent,
150 + );
151 + state = reduceProgress(
152 + state,
153 + map({
154 + type: "tool_execution_end",
155 + toolCallId: "custom-1",
156 + toolName: "custom_tool",
157 + result,
158 + isError: false,
159 + }) as UltraProgressEvent,
160 + );
162 + const item = state.transcriptItems[AGENT]?.[0];
163 + expect(item).toMatchObject({
164 + kind: "tool",
165 + result,
166 + fallback: "Result · ok=true · files=[2] · metadata={1}",
167 + });
168 + });
170 + it.each([
171 + [null, "Result · null"],
172 + [true, "Result · true"],
173 + [42, "Result · 42"],
174 + [[1, 2, 3], "Result · 3 items"],
176 + { content: [{ type: "text", text: "Indexed 3 files\nverbose" }] },
177 + "Result · Indexed 3 files",
178 + ],
179 + ])("renders JSON result %j correctly", (result, expected) => {
180 + const state = reduceProgress(
181 + initialReducerState(),
182 + map({
183 + type: "tool_execution_end",
184 + toolCallId: "custom-1",
185 + toolName: "custom_tool",
186 + result,
187 + isError: false,
188 + }) as UltraProgressEvent,
189 + );
190 + expect(state.transcriptItems[AGENT]?.[0]).toMatchObject({
191 + fallback: expected,
192 + result,
193 + });
194 + });
196 + it("maps tool execution updates as partial results", () => {
197 + const partialResult = {
198 + content: [{ type: "text", text: "first chunk" }],
199 + };
200 + expect(
201 + map({
202 + type: "tool_execution_update",
203 + toolCallId: "t1",
204 + toolName: "read",
205 + args: { path: "a.ts" },
206 + partialResult,
207 + }),
208 + ).toMatchObject({
209 + kind: "action",
210 + toolCallId: "t1",
211 + toolName: "read",
212 + result: partialResult,
213 + isPartial: true,
214 + });
215 + });
103 217 it("returns null for events that do not map", () => {
104 218 expect(map({ type: "turn_start" })).toBeNull();
105 219 expect(map({ type: "message_start", message: {} })).toBeNull();
331 - expect(s.rows[0].action).toContain("bash $ git status --short");
445 + expect(s.rows[0].action).toBe("Running git status --short");
332 446 expect(s.rows[0].action).not.toContain("RAW RESULT");
449 + it("prefers an LLM action summary over raw tool args", () => {
450 + const s = fold([
451 + pe({
452 + agentId: "a",
453 + phase: "p1",
454 + kind: "action",
455 + toolName: "read",
456 + args: { path: "extensions/ultra/progress.ts" },
457 + actionSummary: "Inspecting workflow progress state",
458 + }),
459 + ]);
461 + expect(s.rows[0].action).toBe("Inspecting workflow progress state");
462 + expect(s.rows[0].action).not.toContain("progress.ts");
463 + });
335 465 it("summarizes structured_output rows instead of dumping JSON", () => {
336 466 const s = fold([
504 634 expect(s1.rows[0]).not.toBe(s0.rows[0]); // changed row is fresh
637 + it("merges tool start and end into one transcript item by tool call id", () => {
638 + const result = {
639 + content: [{ type: "text", text: "file contents" }],
640 + details: { lines: 1 },
641 + };
642 + const s = fold([
643 + pe({
644 + agentId: "a",
645 + phase: "p1",
646 + kind: "action",
647 + toolCallId: "tool-1",
648 + toolName: "read",
649 + args: { path: "file.ts" },
650 + }),
651 + pe({
652 + agentId: "a",
653 + phase: "p1",
654 + kind: "action",
655 + toolCallId: "tool-1",
656 + toolName: "read",
657 + result: { content: [{ type: "text", text: "partial" }] },
658 + isPartial: true,
659 + }),
660 + pe({
661 + agentId: "a",
662 + phase: "p1",
663 + kind: "action",
664 + toolCallId: "tool-1",
665 + toolName: "read",
666 + result,
667 + isError: false,
668 + isPartial: false,
669 + }),
670 + ]);
672 + expect(s.transcriptItems.a).toHaveLength(1);
673 + expect(s.transcriptItems.a[0]).toMatchObject({
674 + kind: "tool",
675 + toolCallId: "tool-1",
676 + toolName: "read",
677 + args: { path: "file.ts" },
678 + result,
679 + isError: false,
680 + isPartial: false,
681 + });
682 + });
507 684 it("end-to-end: mapped session events fold into the expected board state", () => {
508 685 const raw = [
509 686 { type: "agent_start" },
536 713 phase: "review",
537 714 status: "done",
539 - expect(s.rows[0].action).toContain("grep");
716 + expect(s.rows[0].action).toBe("Searching x");
540 717 expect(s.transcripts.g0).toContain("done.");
541 718 expect(s).toMatchObject({ total: 1, done: 1 });

extensions/ultra/__tests__/prompt.test.ts

Modified
30 30 const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
32 32 expect(prompt).toContain("# Authoring ultra workflows");
33 - expect(prompt).toContain("/ultra <name>");
33 + expect(prompt).toContain("/ultra run <name>");
34 34 expect(prompt).not.toContain("name: authoring-workflows");
35 + expect(reference).toContain("/ultra run <name>");
35 36 expect(reference).toContain("Every step returns a JSON object");
37 + expect(prompt).toContain("Ultra refuses to start the workflow");
38 + expect(reference).toContain("Ultra refuses to start the workflow");
39 + const readme = fs.readFileSync(README_PATH, "utf8");
40 + expect(readme).toContain("`/ultra exec <instructions>`");
41 + expect(readme).toContain("`/ultra run <name> [args]`");
42 + expect(readme).not.toContain("`/ultra history`");
43 + expect(readme).not.toContain("`/ultra runs`");
38 46 it("exposes invocation-critical semantics through the tool schema", () => {
41 49 "exact key in top-level schemas",
42 50 "{PHASE.results[].FIELD[]}",
43 51 "single brace-wrapped selector",
52 + "Every listed tool must be available",
44 53 "read, grep, find, ls, bash",
54 + "web_search, web_fetch, and web_map",
45 55 "forbid mutation",
47 57 expect(toolGuidance).toContain(rule);

extensions/ultra/__tests__/research.test.ts

Added
new file mode 100644
1 +import * as fs from "node:fs";
2 +import * as os from "node:os";
3 +import * as path from "node:path";
4 +import { fileURLToPath } from "node:url";
5 +import { describe, expect, it } from "vitest";
6 +import { resolveSelector } from "../interp.ts";
7 +import { compileSchema, discoverWorkflows, parseWorkflow } from "../spec.ts";
9 +const here = path.dirname(fileURLToPath(import.meta.url));
10 +const RESEARCH_PATH = path.join(here, "..", "workflows", "research.json");
11 +const READ_ONLY_TOOLS = new Set([
12 + "read",
13 + "grep",
14 + "find",
15 + "ls",
16 + "bash",
17 + "web_search",
18 + "web_fetch",
19 + "web_map",
20 +]);
22 +function loadRaw(): string {
23 + return fs.readFileSync(RESEARCH_PATH, "utf8");
26 +describe("research shipped workflow", () => {
27 + it("parses the shipped five-phase research pipeline", () => {
28 + const spec = parseWorkflow(loadRaw());
29 + expect(spec.name).toBe("research");
30 + expect(spec.args?.params).toEqual(["question"]);
31 + expect(spec.phases.map((phase) => phase.id)).toEqual([
32 + "plan",
33 + "gather",
34 + "audit",
35 + "verify",
36 + "report",
37 + ]);
38 + expect(spec.return).toBe("{report.results}");
39 + });
41 + it("compiles every named output schema", () => {
42 + const spec = parseWorkflow(loadRaw());
43 + for (const schema of Object.values(spec.schemas ?? {}))
44 + expect(compileSchema(schema)).toBeDefined();
45 + });
47 + it("fans out breadth, audits centrally, then fans out targeted verification", () => {
48 + const spec = parseWorkflow(loadRaw());
49 + const phase = (id: string) =>
50 + spec.phases.find((candidate) => candidate.id === id);
52 + expect(phase("gather")?.over).toBe("{plan.results[].angles[]}");
53 + expect(phase("audit")?.step.prompt).toContain("{gather.results}");
54 + expect(phase("verify")?.when).toBe(
55 + "{audit.results[].verification_tasks[]}",
56 + );
57 + expect(phase("verify")?.over).toBe(
58 + "{audit.results[].verification_tasks[]}",
59 + );
60 + expect(phase("report")?.step.prompt).toContain("{verify.results}");
62 + const plan = spec.schemas?.ResearchPlan as {
63 + properties?: { angles?: { maxItems?: number } };
64 + };
65 + const audit = spec.schemas?.Audit as {
66 + properties?: { verification_tasks?: { maxItems?: number } };
67 + };
68 + expect(plan.properties?.angles?.maxItems).toBe(4);
69 + expect(audit.properties?.verification_tasks?.maxItems).toBe(3);
70 + expect(phase("gather")?.step.prompt).toContain(
71 + "maxChars no greater than 8000",
72 + );
73 + expect(phase("verify")?.step.prompt).toContain(
74 + "at most 3 substantive sources",
75 + );
76 + });
78 + it("keeps parallel web research read-only and evidence-driven", () => {
79 + const spec = parseWorkflow(loadRaw());
80 + for (const phase of spec.phases) {
81 + if (phase.kind !== "fanout") continue;
82 + for (const tool of phase.step.tools ?? [])
83 + expect(READ_ONLY_TOOLS.has(tool)).toBe(true);
84 + expect(phase.step.prompt).toMatch(/do not modify/i);
85 + expect(phase.step.tools).toEqual(
86 + expect.arrayContaining(["web_search", "web_fetch"]),
87 + );
88 + }
89 + });
91 + it("assigns cheap breadth and strong planning, audit, verification, and synthesis", () => {
92 + const spec = parseWorkflow(loadRaw());
93 + expect(
94 + Object.fromEntries(
95 + spec.phases.map((phase) => [
96 + phase.id,
97 + [phase.step.model, phase.step.thinkingLevel],
98 + ]),
99 + ),
100 + ).toEqual({
101 + plan: ["big", "high"],
102 + gather: ["small", "medium"],
103 + audit: ["big", "high"],
104 + verify: ["big", "high"],
105 + report: ["big", "high"],
106 + });
107 + });
109 + it("requires traceable claims and a decision-ready final report", () => {
110 + const spec = parseWorkflow(loadRaw());
111 + const report = spec.schemas?.ResearchReport as {
112 + required?: string[];
113 + properties?: Record<string, unknown>;
114 + };
115 + for (const field of [
116 + "answer_status",
117 + "executive_summary",
118 + "analysis",
119 + "findings",
120 + "contradictions",
121 + "gaps",
122 + "recommendations",
123 + "sources",
124 + ])
125 + expect(report.required).toContain(field);
126 + expect(report.properties).toHaveProperty("findings");
128 + const value = {
129 + answer_status: "fully answered",
130 + executive_summary: "Answer",
131 + analysis: "Analysis",
132 + findings: [],
133 + contradictions: [],
134 + gaps: [],
135 + recommendations: [],
136 + sources: [],
137 + };
138 + expect(
139 + resolveSelector(spec.return ?? "", { results: { report: [value] } }),
140 + ).toEqual([value]);
141 + });
143 + it("is discoverable as a bundled workflow", () => {
144 + const cwd = fs.mkdtempSync(path.join(os.tmpdir(), "ultra-cwd-"));
145 + const home = fs.mkdtempSync(path.join(os.tmpdir(), "ultra-home-"));
146 + try {
147 + const research = discoverWorkflows(cwd, { home }).find(
148 + (workflow) => workflow.name === "research",
149 + );
150 + expect(research?.source).toBe("bundled");
151 + } finally {
152 + fs.rmSync(cwd, { recursive: true, force: true });
153 + fs.rmSync(home, { recursive: true, force: true });
155 + });
156 +});

extensions/ultra/__tests__/runner.test.ts

Modified
178 178 partial: {},
181 + emit({
182 + type: "tool_execution_start",
183 + toolCallId: "t2",
184 + toolName: STRUCTURED_OUTPUT_TOOL_NAME,
185 + args: params,
186 + });
181 187 const result = await callStructured(params);
182 188 emit({
183 189 type: "tool_execution_end",
789 795 // the scripted sequence maps to ultra events, each tagged with the agent id
790 796 const kinds = events.map((e) => e.kind);
791 - expect(kinds).toContain("action"); // tool_execution_start/end
797 + expect(kinds).toContain("action"); // user-facing tool execution
798 + expect(
799 + events.some(
800 + (event) =>
801 + event.kind === "action" &&
802 + event.toolName === STRUCTURED_OUTPUT_TOOL_NAME,
803 + ),
804 + ).toBe(false);
792 805 expect(kinds).toContain("delta"); // text_delta
793 806 expect(kinds).toContain("end"); // agent_end
794 807 for (const e of events) expect(e.agentId).toBe("review#7");
810 823 expect(harness.captured.spies?.dispose).toHaveBeenCalledTimes(1);
826 + it("flushes and applies a fast agent's latest LLM summary", async () => {
827 + let resolveSummary!: (summary: string) => void;
828 + const actionSummarizer = Object.assign(
829 + vi.fn(
830 + () =>
831 + new Promise<string>((resolve) => {
832 + resolveSummary = resolve;
833 + }),
834 + ),
836 + finish: vi.fn(() => resolveSummary("Reading the target source file")),
837 + cancel: vi.fn(),
838 + dispose: vi.fn(),
839 + },
840 + );
841 + const events: UltraProgressEvent[] = [];
842 + const harness = makeSessionHarness({
843 + drive: async ({ emit, callStructured }) => {
844 + emit({ type: "agent_start" });
845 + emit({
846 + type: "tool_execution_start",
847 + toolCallId: "read-1",
848 + toolName: "read",
849 + args: { path: "a.ts" },
850 + });
851 + await callStructured({ verdict: "ok" });
852 + emit({ type: "agent_end", messages: [], willRetry: false });
853 + },
854 + });
856 + await runStep(
857 + toolStep,
858 + makeCtx(),
859 + makeDrivers({ sessionFactory: harness.factory }),
860 + baseOpts({
861 + actionSummarizer,
862 + onProgress: (event) => events.push(event),
863 + }),
864 + );
865 + await new Promise((resolve) => setTimeout(resolve, 0));
867 + expect(actionSummarizer).toHaveBeenCalledWith({
868 + agentId: "review#0",
869 + toolName: "read",
870 + args: { path: "a.ts" },
871 + taskSummary: toolStep.summary,
872 + signal: undefined,
873 + });
874 + expect(actionSummarizer.finish).toHaveBeenCalledWith("review#0");
875 + const actions = events.filter(
876 + (event) => event.kind === "action" && event.toolCallId === "read-1",
877 + );
878 + expect(actions).toHaveLength(2);
879 + expect(actions[0]).toMatchObject({ text: expect.stringContaining("a.ts") });
880 + expect(actions[0].actionSummary).toBeUndefined();
881 + expect(actions[1]).toMatchObject({
882 + actionSummary: "Reading the target source file",
883 + });
884 + });
886 + it("drops an action summary superseded by a newer tool call", async () => {
887 + const resolvers = new Map<string, (summary: string) => void>();
888 + const actionSummarizer = vi.fn(
889 + ({ toolName }: { toolName: string }) =>
890 + new Promise<string>((resolve) => resolvers.set(toolName, resolve)),
891 + );
892 + const events: UltraProgressEvent[] = [];
893 + const harness = makeSessionHarness({
894 + drive: async ({ emit, callStructured }) => {
895 + emit({ type: "agent_start" });
896 + emit({
897 + type: "tool_execution_start",
898 + toolCallId: "read-1",
899 + toolName: "read",
900 + args: { path: "old.ts" },
901 + });
902 + emit({
903 + type: "tool_execution_start",
904 + toolCallId: "grep-1",
905 + toolName: "grep",
906 + args: { pattern: "new" },
907 + });
908 + resolvers.get("read")?.("Reading stale source");
909 + resolvers.get("grep")?.("Finding the new implementation");
910 + await new Promise((resolve) => setTimeout(resolve, 0));
911 + await callStructured({ verdict: "ok" });
912 + emit({ type: "agent_end", messages: [], willRetry: false });
913 + },
914 + });
916 + await runStep(
917 + toolStep,
918 + makeCtx(),
919 + makeDrivers({ sessionFactory: harness.factory }),
920 + baseOpts({
921 + actionSummarizer,
922 + onProgress: (event) => events.push(event),
923 + }),
924 + );
926 + expect(
927 + events.some((event) => event.actionSummary === "Reading stale source"),
928 + ).toBe(false);
929 + expect(
930 + events.some(
931 + (event) => event.actionSummary === "Finding the new implementation",
932 + ),
933 + ).toBe(true);
934 + });
813 936 it("lets a stalled session run until it completes", async () => {
814 937 let release!: () => void;
815 938 const harness = makeSessionHarness({

extensions/ultra/__tests__/settings.test.ts

Modified
13 13 concurrency: 5,
14 14 maxRetries: 2,
15 15 subagentExtensions: "all",
16 + subagentSkills: "all",
16 17 modelTiers: {},
18 + actionSummarizerModel: null,
57 +describe("subagentSkills setting", () => {
58 + it('defaults to "all" and supports explicit opt-out', () => {
59 + expect(ULTRA_DEFAULTS.subagentSkills).toBe("all");
60 + expect(pickUltra({ ultra: { subagentSkills: "none" } })).toEqual({
61 + subagentSkills: "none",
62 + });
63 + expect(pickUltra({ ultra: { subagentSkills: "all" } })).toEqual({
64 + subagentSkills: "all",
65 + });
66 + expect(pickUltra({ ultra: { subagentSkills: true } })).toEqual({});
67 + });
69 + it("merges later valid values and ignores invalid ones", () => {
70 + expect(
71 + mergeUltraSettings({ subagentSkills: "none" }, { subagentSkills: "all" })
72 + .subagentSkills,
73 + ).toBe("all");
74 + expect(
75 + mergeUltraSettings({ subagentSkills: false as never }).subagentSkills,
76 + ).toBe("all");
77 + });
78 +});
55 80 describe("pickUltra", () => {
56 81 it("extracts only correctly-typed known keys from the `ultra` block", () => {
57 82 expect(
60 85 concurrency: 8,
61 86 maxRetries: 1,
62 87 enabled: false,
88 + actionSummarizerModel: " openai/gpt-5-mini ",
63 89 modelTiers: {
64 90 small: "openai/gpt-5-mini:low",
65 91 medium: "anthropic/claude-sonnet-4",
72 98 enabled: false,
73 99 concurrency: 8,
74 100 maxRetries: 1,
101 + actionSummarizerModel: "openai/gpt-5-mini",
75 102 modelTiers: {
76 103 small: "openai/gpt-5-mini:low",
77 104 medium: "anthropic/claude-sonnet-4",
116 143 concurrency: 9,
117 144 maxRetries: 4,
118 145 subagentExtensions: "all",
146 + subagentSkills: "all",
119 147 modelTiers: {
120 148 small: "openai/gpt-5-mini",
121 149 big: "openai/gpt-5",
151 + actionSummarizerModel: null,
139 168 global: {
140 169 ultra: {
141 170 concurrency: 10,
171 + actionSummarizerModel: "openai/gpt-5-mini",
142 172 modelTiers: { small: "openai/gpt-5-mini" },
155 185 concurrency: 10,
156 186 maxRetries: 9,
157 187 subagentExtensions: "all",
188 + subagentSkills: "all",
158 189 modelTiers: {
159 190 small: "openai/gpt-5-mini",
160 191 big: "openai/gpt-5",
193 + actionSummarizerModel: "openai/gpt-5-mini",
197 + it("allows project null to disable the global action summarizer", () => {
198 + expect(
199 + loadUltraSettings({
200 + global: {
201 + ultra: { actionSummarizerModel: "openai/gpt-5-mini" },
202 + },
203 + project: { ultra: { actionSummarizerModel: null } },
204 + }).actionSummarizerModel,
205 + ).toBeNull();
206 + });
165 208 it("falls back to defaults when no source provides values", () => {
166 209 expect(loadUltraSettings({})).toEqual(ULTRA_DEFAULTS);

extensions/ultra/__tests__/spec.test.ts

Modified
2 2 import * as fs from "node:fs";
3 3 import * as os from "node:os";
4 4 import * as path from "node:path";
5 5 import { afterAll, beforeAll, describe, expect, it } from "vitest";
6 -import { compileSchema, discoverWorkflows, parseWorkflow } from "../spec";
6 +import {
7 + compileSchema,
8 + discoverWorkflows,
9 + missingWorkflowTools,
10 + parseWorkflow,
11 +} from "../spec";
8 13 const FINDINGS_SCHEMA = {
9 14 type: "object",
219 +describe("missingWorkflowTools", () => {
220 + it("returns every distinct unavailable step tool in stable order", () => {
221 + const spec = parseWorkflow(validSpec());
222 + expect(missingWorkflowTools(spec, ["read", "bash", "grep"])).toEqual([
223 + "find",
224 + "ls",
225 + ]);
226 + expect(
227 + missingWorkflowTools(spec, ["read", "bash", "grep", "find", "ls"]),
228 + ).toEqual([]);
229 + });
230 +});
214 232 describe("compileSchema", () => {
215 233 it("compiles a named schema from the schemas map into a working validator", () => {
216 234 const v = compileSchema(FINDINGS_SCHEMA);

extensions/ultra/__tests__/ui.test.ts

Modified
1 +import { stripVTControlCharacters } from "node:util";
1 2 import type {
2 3 Theme,
3 4 ToolRenderResultOptions,
4 5 } from "@earendil-works/pi-coding-agent";
5 6 import { initTheme } from "@earendil-works/pi-coding-agent";
7 +import type { TUI } from "@earendil-works/pi-tui";
6 8 import { beforeAll, describe, expect, it, vi } from "vitest";
7 9 import {
8 10 type AgentControls,
36 38 now?: () => number;
37 39 throttleMs?: number;
38 40 onRunAbort?: () => void;
41 + toolUi?: boolean;
42 + toolsExpanded?: () => boolean;
43 + hideToolPreviews?: boolean;
44 + isToolsExpandKey?: (data: string) => boolean;
45 + onToolsExpand?: () => void;
40 47 const requestRender = vi.fn();
41 48 const host: OverlayHost = {
48 55 onRunAbort: opts.onRunAbort ?? vi.fn(),
49 56 now: opts.now,
50 57 throttleMs: opts.throttleMs,
58 + toolUi: opts.toolUi ? (host as unknown as TUI) : undefined,
59 + toolsExpanded: opts.toolsExpanded,
60 + hideToolPreviews: opts.hideToolPreviews,
61 + isToolsExpandKey: opts.isToolsExpandKey,
62 + onToolsExpand: opts.onToolsExpand,
63 + cwd: opts.toolUi ? process.cwd() : undefined,
52 65 return { overlay, requestRender, host };
705 718 expect(overlay.render(60).length).toBeLessThanOrEqual(9);
721 + it("keeps one native Pi tool row through start, result, and expansion", () => {
722 + let expanded = false;
723 + const { overlay } = makeOverlay({
724 + rows: 80,
725 + toolUi: true,
726 + toolsExpanded: () => expanded,
727 + isToolsExpandKey: (data) => data === "EXPAND",
728 + onToolsExpand: () => {
729 + expanded = !expanded;
730 + },
731 + });
732 + overlay.handleProgress(pe({ agentId: "a", phase: "p", kind: "start" }));
733 + overlay.handleProgress(
734 + pe({
735 + agentId: "a",
736 + phase: "p",
737 + kind: "action",
738 + toolCallId: "tool-1",
739 + toolName: "read",
740 + args: { path: "extensions/ultra/index.ts" },
741 + }),
742 + );
744 + let out = stripVTControlCharacters(overlay.render(100).join("\n"));
745 + expect(out).toContain("extensions/ultra/index.ts");
746 + expect(out).not.toContain("read ...");
748 + overlay.handleProgress(
749 + pe({
750 + agentId: "a",
751 + phase: "p",
752 + kind: "action",
753 + toolCallId: "tool-1",
754 + toolName: "read",
755 + result: {
756 + content: [{ type: "text", text: "PARTIAL-RESULT" }],
757 + },
758 + isPartial: true,
759 + }),
760 + );
761 + out = stripVTControlCharacters(overlay.render(100).join("\n"));
762 + expect(out).not.toContain("PARTIAL-RESULT");
764 + overlay.handleInput("EXPAND");
765 + expect(expanded).toBe(true);
766 + out = stripVTControlCharacters(overlay.render(100).join("\n"));
767 + expect(out).toContain("PARTIAL-RESULT");
769 + overlay.handleProgress(
770 + pe({
771 + agentId: "a",
772 + phase: "p",
773 + kind: "action",
774 + toolCallId: "tool-1",
775 + toolName: "read",
776 + result: {
777 + content: [{ type: "text", text: "NATIVE-RESULT" }],
778 + },
779 + isError: false,
780 + isPartial: false,
781 + }),
782 + );
783 + out = stripVTControlCharacters(overlay.render(100).join("\n"));
784 + expect(out).toContain("NATIVE-RESULT");
785 + expect(out).not.toContain("PARTIAL-RESULT");
786 + expect(out).not.toContain("read ...");
787 + });
789 + it("hides collapsed tool previews when action summaries are enabled", () => {
790 + let expanded = false;
791 + const { overlay } = makeOverlay({
792 + rows: 80,
793 + toolUi: true,
794 + hideToolPreviews: true,
795 + toolsExpanded: () => expanded,
796 + isToolsExpandKey: (data) => data === "EXPAND",
797 + onToolsExpand: () => {
798 + expanded = !expanded;
799 + },
800 + });
801 + overlay.handleProgress(pe({ agentId: "a", phase: "p", kind: "start" }));
802 + overlay.handleProgress(
803 + pe({
804 + agentId: "a",
805 + phase: "p",
806 + kind: "action",
807 + toolCallId: "tool-1",
808 + toolName: "read",
809 + args: { path: "HIDDEN-PREVIEW.ts" },
810 + actionSummary: "Inspecting hidden preview",
811 + }),
812 + );
814 + let out = stripVTControlCharacters(overlay.render(100).join("\n"));
815 + expect(out).not.toContain("HIDDEN-PREVIEW.ts");
816 + overlay.handleInput("EXPAND");
817 + out = stripVTControlCharacters(overlay.render(100).join("\n"));
818 + expect(out).toContain("HIDDEN-PREVIEW.ts");
819 + });
708 821 it("renders transcript markdown with Pi's markdown renderer", () => {
709 822 const { overlay } = makeOverlay({ rows: 24 });
710 823 overlay.handleProgress(pe({ agentId: "a", phase: "p", kind: "start" }));

extensions/ultra/__tests__/wire.test.ts

Modified
2 2 import { describe, expect, it, vi } from "vitest";
3 3 import type { AgentProgressEvent, UltraProgressEvent } from "../progress.ts";
4 4 import type { DiscoveredWorkflow, WorkflowSpec } from "../spec.ts";
5 5 import {
6 + completeUltraCommand,
6 7 completeWorkflowNames,
7 - formatBackgroundAcceptedMessage,
8 - formatBackgroundResultMessage,
9 8 formatUltraResultMarkdown,
10 9 formatWorkflowResultMessage,
11 10 makeToolProgressSink,
12 11 mapPositionalArgs,
13 12 parseCommandLine,
13 + parseUltraCommand,
14 14 resolveWorkflowInput,
15 15 } from "../wire.ts";
151 151 expect(markdown).not.toContain("```json");
154 + it("renders synthesized review analysis and concrete recommendations", () => {
155 + const markdown = formatUltraResultMarkdown({
156 + workflow: "review",
157 + phases: [{ id: "report", ran: 1, ok: 1, dropped: 0 }],
158 + phaseResults: {},
159 + phaseFailures: {},
160 + result: [
162 + analysis:
163 + "The implementation is cohesive, but one boundary check is missing.",
164 + findings: [
166 + title: "Missing boundary check",
167 + file: "src/parser.ts",
168 + why: "Malformed input reaches the decoder.",
169 + severity: "high",
170 + },
171 + ],
172 + recommendations: [
173 + "Reject malformed lengths before decoding.",
174 + "Add the malformed-input regression test, then rerun review.",
175 + ],
176 + },
177 + ],
178 + steered: false,
179 + });
181 + expect(markdown).toContain("## Analysis");
182 + expect(markdown).toContain(
183 + "The implementation is cohesive, but one boundary check is missing.",
184 + );
185 + expect(markdown).toContain("## Verified findings");
186 + expect(markdown).toContain("Missing boundary check");
187 + expect(markdown).toContain("## Recommendations");
188 + expect(markdown).toContain("Reject malformed lengths before decoding.");
189 + expect(markdown).not.toContain("Continue with the final results above.");
190 + expect(markdown).not.toContain("```json");
191 + });
193 + it("renders a cited research report instead of dumping JSON", () => {
194 + const result = {
195 + workflow: "research",
196 + phases: [{ id: "report", ran: 1, ok: 1, dropped: 0 }],
197 + phaseResults: {},
198 + phaseFailures: {},
199 + result: [
201 + answer_status: "partially answered",
202 + executive_summary:
203 + "The evidence supports the main claim with one caveat.",
204 + analysis: "Primary and independent sources converge.",
205 + findings: [
207 + claim: "The main claim is supported",
208 + evidence: "Two independent primary sources agree.",
209 + confidence: "high",
210 + citations: ["https://example.com/primary"],
211 + },
212 + ],
213 + contradictions: ["One older source disagrees."],
214 + gaps: ["No current regional data."],
215 + recommendations: ["Collect regional data before deciding."],
216 + sources: [
218 + title: "Primary source",
219 + url: "https://example.com/primary",
220 + source_type: "primary",
221 + published_at: "2026-08-01",
222 + },
223 + ],
224 + },
225 + ],
226 + steered: false,
227 + };
229 + const markdown = formatUltraResultMarkdown(result);
230 + expect(markdown).toContain("## Answer");
231 + expect(markdown).toContain("Status: partially answered");
232 + expect(markdown).toContain("## Findings");
233 + expect(markdown).toContain("Sources: https://example.com/primary");
234 + expect(markdown).toContain("## Contradictions");
235 + expect(markdown).toContain("## Research gaps");
236 + expect(markdown).toContain("## Sources");
237 + expect(markdown).not.toContain("```json");
238 + expect(formatWorkflowResultMessage(result)).toBe(markdown);
239 + });
154 241 it("renders aggregate token and cost usage when present", () => {
155 242 const markdown = formatUltraResultMarkdown({
156 243 workflow: "review",
191 278 // formatWorkflowResultMessage — session/LLM payload
192 279 // ---------------------------------------------------------------------------
194 -describe("background result messages", () => {
195 - it("formats start and completion messages for detached workflow runs", () => {
196 - const result = {
197 - workflow: "review",
198 - phases: [{ id: "p", ran: 1, ok: 1, dropped: 0 }],
199 - phaseResults: {},
200 - phaseFailures: {},
201 - result: [{ title: "bug" }],
202 - steered: false,
203 - tokenUsage: {
204 - input: 1,
205 - output: 2,
206 - total: 3,
207 - cacheRead: 0,
208 - cacheWrite: 0,
209 - cost: 0,
210 - },
211 - };
213 - expect(formatBackgroundAcceptedMessage("run-1", "review")).toBe(
214 - 'ultra background run "review" started: run-1',
215 - );
216 - expect(formatBackgroundResultMessage("run-1", result)).toContain(
217 - 'ultra background run "review" complete: run-1',
218 - );
219 - expect(formatBackgroundResultMessage("run-1", result)).toContain(
220 - "## Verified findings\n- **bug**",
221 - );
222 - });
223 -});
225 281 describe("formatWorkflowResultMessage", () => {
226 282 it("formats a human summary so /ultra command results enter the session", () => {
227 283 const content = formatWorkflowResultMessage({
529 +describe("completeUltraCommand", () => {
530 + const list = [
531 + discovered("review", "bundled"),
532 + discovered("repair", "project"),
533 + ];
535 + it("completes only subcommands at the top level", () => {
536 + expect(completeUltraCommand(list, "")?.map((item) => item.value)).toEqual([
537 + "exec",
538 + "run",
539 + ]);
540 + expect(completeUltraCommand(list, "r")?.map((item) => item.value)).toEqual([
541 + "run",
542 + ]);
543 + expect(completeUltraCommand(list, "h")).toBeNull();
544 + });
546 + it("completes workflow names only below run", () => {
547 + expect(
548 + completeUltraCommand(list, "run re")?.map((item) => item.value),
549 + ).toEqual(["run review", "run repair"]);
550 + expect(completeUltraCommand(list, "exec re")).toBeNull();
551 + });
552 +});
473 554 // ---------------------------------------------------------------------------
474 555 // resolveWorkflowInput — name (persisted) vs inline spec (ephemeral)
475 556 // ---------------------------------------------------------------------------
547 628 it("validates a named workflow's spec, surfacing parseWorkflow's error on a malformed saved file", () => {
548 629 // Discovery stores the raw JSON.parse output cast to WorkflowSpec — a saved
549 630 // file can be JSON-valid with a `name` yet structurally malformed (no
550 - // `phases`). Resolving by name must run parseWorkflow so `/ultra <name>`
631 + // `phases`). Resolving by name must run parseWorkflow so `/ultra run <name>`
551 632 // fails fast with the TypeBox path error, not a cryptic mid-run crash.
552 633 const malformed: DiscoveredWorkflow = {
553 634 name: "broken",
564 645 // ---------------------------------------------------------------------------
565 -// parseCommandLine — `/ultra <name> [json-args]`
646 +// command parsing
566 647 // ---------------------------------------------------------------------------
649 +describe("parseUltraCommand", () => {
650 + it("routes every feature through an explicit subcommand", () => {
651 + expect(parseUltraCommand(" ")).toEqual({ subcommand: "activate" });
652 + expect(parseUltraCommand("exec fix the race")).toEqual({
653 + subcommand: "exec",
654 + instructions: "fix the race",
655 + });
656 + expect(parseUltraCommand("run review main")).toEqual({
657 + subcommand: "run",
658 + invocation: "review main",
659 + });
660 + });
662 + it("rejects missing arguments and unknown commands", () => {
663 + for (const input of [
664 + "exec",
665 + "run",
666 + "review main",
667 + "runs",
668 + "history",
669 + "history extra",
670 + ]) {
671 + expect(parseUltraCommand(input)).toEqual({ subcommand: "invalid" });
673 + });
674 +});
568 676 describe("parseCommandLine", () => {
569 677 it("returns {} for an empty line", () => {
570 678 expect(parseCommandLine(" ")).toEqual({});
597 705 // ---------------------------------------------------------------------------
598 -// mapPositionalArgs — bare `/ultra <name> <tok...>` → declared args.params
706 +// mapPositionalArgs — bare `/ultra run <name> <tok...>` → declared args.params
599 707 // ---------------------------------------------------------------------------
601 709 describe("mapPositionalArgs", () => {

extensions/ultra/action-summarizer.ts

Added
new file mode 100644
1 +import type {
2 + Model,
3 + ProviderHeaders,
4 + UserMessage,
5 +} from "@earendil-works/pi-ai";
6 +import { complete } from "@earendil-works/pi-ai/compat";
7 +import type { ExtensionContext } from "@earendil-works/pi-coding-agent";
9 +export interface ActionSummaryInput {
10 + agentId: string;
11 + toolName: string;
12 + args: unknown;
13 + taskSummary: string;
14 + signal?: AbortSignal;
17 +export type ActionSummarizer = ((
18 + input: ActionSummaryInput,
19 +) => Promise<string | undefined>) & {
20 + finish(agentId: string): void;
21 + cancel(agentId: string): void;
22 + dispose(): void;
23 +};
25 +type GenerateActionSummary = typeof complete;
27 +type TimerScheduler = {
28 + setTimeout(callback: () => void, delayMs: number): unknown;
29 + clearTimeout(handle: unknown): void;
30 +};
32 +type ActionSummarizerAuth = {
33 + model: Model<any>;
34 + apiKey?: string;
35 + headers?: ProviderHeaders;
36 +};
38 +type Job = {
39 + input: ActionSummaryInput;
40 + resolve(summary: string | undefined): void;
41 + controller?: AbortController;
42 + cancelled?: boolean;
43 +};
45 +type ActionSummarizerOptions = {
46 + generate?: GenerateActionSummary;
47 + scheduler?: TimerScheduler;
48 +};
50 +const MAX_INPUT_CHARS = 1_500;
51 +const MAX_SUMMARY_CHARS = 80;
52 +const MAX_TOKENS = 40;
53 +const TIMEOUT_MS = 2_000;
54 +const COALESCE_MS = 350;
56 +const SYSTEM_PROMPT = `Summarize one coding-agent tool action as a present-progress phrase.
57 +Describe intent, not tool mechanics.
58 +Start with an -ing verb.
59 +Return exactly one plain-text phrase under 60 characters.
60 +Do not use tool names, JSON, markdown, bullets, quotes, or a trailing period.`;
62 +const ANSI_RE =
63 + /\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~]|\][^\x07]*(?:\x07|\x1B\\))/g;
64 +const CONTROL_RE = /[\x00-\x1F\x7F-\x9F]/g;
66 +const defaultScheduler: TimerScheduler = {
67 + setTimeout(callback, delayMs) {
68 + return setTimeout(callback, delayMs);
69 + },
70 + clearTimeout(handle) {
71 + clearTimeout(handle as ReturnType<typeof setTimeout>);
72 + },
73 +};
75 +function parseModelRef(
76 + value: string,
77 +): { provider: string; id: string } | undefined {
78 + const separator = value.indexOf("/");
79 + if (separator <= 0 || separator === value.length - 1) return undefined;
80 + return {
81 + provider: value.slice(0, separator),
82 + id: value.slice(separator + 1),
83 + };
86 +function cap(text: string, maxChars: number): string {
87 + const chars = Array.from(text);
88 + return chars.length <= maxChars
89 + ? text
90 + : `${chars.slice(0, maxChars - 1).join("")}…`;
93 +function stringify(value: unknown): string {
94 + if (typeof value === "string") return value;
95 + try {
96 + return JSON.stringify(value) ?? String(value);
97 + } catch {
98 + return String(value);
99 + }
102 +function sanitize(text: string): string {
103 + return cap(
104 + text
105 + .replace(ANSI_RE, "")
106 + .replace(CONTROL_RE, " ")
107 + .replace(/\s+/g, " ")
108 + .trim()
109 + .replace(/^["'`]+|["'`.]+$/g, ""),
110 + MAX_SUMMARY_CHARS,
111 + );
114 +function prompt(input: ActionSummaryInput): UserMessage {
115 + return {
116 + role: "user",
117 + content: [
119 + type: "text",
120 + text: [
121 + `Task: ${cap(input.taskSummary, MAX_INPUT_CHARS)}`,
122 + `Tool: ${input.toolName}`,
123 + `Arguments: ${cap(stringify(input.args), MAX_INPUT_CHARS)}`,
124 + ].join("\n"),
125 + },
126 + ],
127 + timestamp: Date.now(),
128 + };
131 +class ActionSummarizerEngine {
132 + private readonly pending = new Map<string, Job>();
133 + private readonly queue: Job[] = [];
134 + private readonly timers = new Map<string, unknown>();
135 + private inFlight: Job | undefined;
136 + private disposed = false;
138 + constructor(
139 + private readonly auth: () => Promise<ActionSummarizerAuth | undefined>,
140 + private readonly generate: GenerateActionSummary,
141 + private readonly scheduler: TimerScheduler,
142 + ) {}
144 + enqueue(input: ActionSummaryInput): Promise<string | undefined> {
145 + if (this.disposed) return Promise.resolve(undefined);
146 + return new Promise((resolve) => {
147 + this.pending.get(input.agentId)?.resolve(undefined);
148 + this.pending.set(input.agentId, { input, resolve });
149 + this.clearTimer(input.agentId);
150 + this.timers.set(
151 + input.agentId,
152 + this.scheduler.setTimeout(() => {
153 + this.timers.delete(input.agentId);
154 + this.flush(input.agentId);
155 + }, COALESCE_MS),
156 + );
157 + });
160 + finish(agentId: string): void {
161 + this.clearTimer(agentId);
162 + this.flush(agentId);
165 + cancel(agentId: string): void {
166 + this.clearTimer(agentId);
167 + const pending = this.pending.get(agentId);
168 + if (pending) {
169 + this.pending.delete(agentId);
170 + pending.resolve(undefined);
172 + for (let i = this.queue.length - 1; i >= 0; i--) {
173 + if (this.queue[i].input.agentId !== agentId) continue;
174 + this.queue.splice(i, 1)[0].resolve(undefined);
176 + if (this.inFlight?.input.agentId === agentId) {
177 + this.inFlight.cancelled = true;
178 + this.inFlight.controller?.abort();
182 + dispose(): void {
183 + this.disposed = true;
184 + const agentIds = new Set([
185 + ...this.pending.keys(),
186 + ...this.queue.map((job) => job.input.agentId),
187 + ...(this.inFlight ? [this.inFlight.input.agentId] : []),
188 + ]);
189 + for (const agentId of agentIds) this.cancel(agentId);
192 + private clearTimer(agentId: string): void {
193 + const timer = this.timers.get(agentId);
194 + if (timer === undefined) return;
195 + this.scheduler.clearTimeout(timer);
196 + this.timers.delete(agentId);
199 + private flush(agentId: string): void {
200 + const job = this.pending.get(agentId);
201 + if (!job) return;
202 + this.pending.delete(agentId);
203 + const queued = this.queue.findIndex(
204 + (entry) => entry.input.agentId === agentId,
205 + );
206 + if (queued < 0) this.queue.push(job);
207 + else {
208 + this.queue[queued].resolve(undefined);
209 + this.queue[queued] = job;
211 + this.pump();
214 + private pump(): void {
215 + if (this.inFlight) return;
216 + const job = this.queue.shift();
217 + if (!job) return;
218 + this.inFlight = job;
219 + void this.run(job)
220 + .then(job.resolve, () => job.resolve(undefined))
221 + .finally(() => {
222 + if (this.inFlight === job) this.inFlight = undefined;
223 + this.pump();
224 + });
227 + private async run(job: Job): Promise<string | undefined> {
228 + const resolved = await this.auth();
229 + if (!resolved || job.cancelled) return undefined;
230 + const controller = new AbortController();
231 + job.controller = controller;
232 + if (job.input.signal?.aborted) controller.abort();
233 + else
234 + job.input.signal?.addEventListener("abort", () => controller.abort(), {
235 + once: true,
236 + });
237 + const response = await this.generate(
238 + resolved.model,
239 + { systemPrompt: SYSTEM_PROMPT, messages: [prompt(job.input)] },
241 + apiKey: resolved.apiKey,
242 + headers: resolved.headers,
243 + maxTokens: MAX_TOKENS,
244 + maxRetries: 0,
245 + cacheRetention: "none",
246 + timeoutMs: TIMEOUT_MS,
247 + signal: controller.signal,
248 + },
249 + );
250 + if (response.stopReason !== "stop") return undefined;
251 + const text = response.content
252 + .filter(
253 + (part): part is { type: "text"; text: string } =>
254 + part.type === "text" && typeof part.text === "string",
256 + .map((part) => part.text)
257 + .join("");
258 + return sanitize(text) || undefined;
262 +export function createActionSummarizer(
263 + ctx: Pick<ExtensionContext, "modelRegistry">,
264 + modelRef: string | null,
265 + options: ActionSummarizerOptions = {},
266 +): ActionSummarizer | undefined {
267 + if (!modelRef) return undefined;
268 + const parsed = parseModelRef(modelRef);
269 + if (!parsed) return undefined;
270 + const model = ctx.modelRegistry.find(parsed.provider, parsed.id);
271 + if (!model) return undefined;
273 + let auth: Promise<ActionSummarizerAuth | undefined> | undefined;
274 + const resolveAuth = () => {
275 + auth ??= Promise.resolve(ctx.modelRegistry.getApiKeyAndHeaders(model)).then(
276 + (result) =>
277 + result.ok
278 + ? {
279 + model,
280 + apiKey: result.apiKey,
281 + headers: result.headers,
283 + : undefined,
284 + );
285 + return auth;
286 + };
287 + const engine = new ActionSummarizerEngine(
288 + resolveAuth,
289 + options.generate ?? complete,
290 + options.scheduler ?? defaultScheduler,
291 + );
292 + const summarize = ((input: ActionSummaryInput) =>
293 + engine.enqueue(input)) as ActionSummarizer;
294 + summarize.finish = (agentId) => engine.finish(agentId);
295 + summarize.cancel = (agentId) => engine.cancel(agentId);
296 + summarize.dispose = () => engine.dispose();
297 + return summarize;
300 +export const __test = {
301 + ActionSummarizerEngine,
302 + parseModelRef,
303 + sanitize,
304 + prompt,
305 + SYSTEM_PROMPT,

extensions/ultra/drivers.ts

Modified
42 42 ) => { reload(): Promise<void> };
43 43 getAgentDir: () => string;
44 44 cwd: string;
45 - /** When true, sub-agents load host extensions (their tools become grantable).
46 - * Omitted/false ⇒ isolated (`noExtensions: true`), today's behaviour. */
45 + /** When true, sub-agents load host extensions (their tools become grantable). */
47 46 loadExtensions?: boolean;
47 + /** When true, sub-agents discover host skills. */
48 + loadSkills?: boolean;
50 51 export function makeSessionFactory(deps: SessionFactoryDeps): SessionFactory {
55 56 customTools,
56 57 sessionManager,
57 58 }) => {
58 - // resourceLoader is REQUIRED for isolation (P1): without it every sub-agent
59 - // loads the host repo's extensions (incl. ultra itself), skills, prompt
60 - // templates, themes, and context files. reload() MUST complete before the
61 - // session is constructed.
59 + // resourceLoader is REQUIRED for controlled discovery (P1). reload() MUST
60 + // complete before the session is constructed.
62 61 const resourceLoader = new deps.DefaultResourceLoader({
63 62 cwd: deps.cwd,
64 63 agentDir: deps.getAgentDir(),
65 - // Only this flag flips with the setting; the rest stay isolated so
66 - // sub-agents never inherit host skills/prompts/themes/context files.
67 64 noExtensions: !deps.loadExtensions,
68 - noSkills: true,
65 + noSkills: !deps.loadSkills,
69 66 noPromptTemplates: true,
70 67 noThemes: true,
71 68 noContextFiles: true,
73 70 await resourceLoader.reload();
72 + const activeTools =
73 + deps.loadSkills && !tools.includes("read") ? [...tools, "read"] : tools;
75 74 const { session } = await deps.createAgentSession({
76 75 cwd: deps.cwd,
77 76 model,
78 77 resourceLoader,
79 78 thinkingLevel,
80 79 sessionManager, // the fresh inMemory manager (#2 — never a parent-branch fork)
81 - tools, // [...step.tools, structuredOutputName] (#3/#5 — filter + active set)
80 + tools: activeTools,
82 81 customTools, // [structuredOutputTool] — built by the runner from the step schema
84 83 return session;

extensions/ultra/engine.ts

Modified
89 89 item?: unknown;
92 -/** The documented aggregate the `workflow` tool / `/ultra` command returns. */
92 +/** The documented aggregate returned by the `workflow` tool and `/ultra run`. */
93 93 export interface WorkflowResult {
94 94 workflow: string;
95 95 phases: PhaseResult[];
117 117 spec: WorkflowSpec,
118 118 args: Record<string, unknown> | undefined,
119 119 deps: EngineDeps,
120 +): Promise<WorkflowResult> {
121 + try {
122 + return await runWorkflowBody(spec, args, deps);
123 + } finally {
124 + deps.stepRunner.dispose?.();
128 +async function runWorkflowBody(
129 + spec: WorkflowSpec,
130 + args: Record<string, unknown> | undefined,
131 + deps: EngineDeps,
120 132 ): Promise<WorkflowResult> {
121 133 const results: Record<string, unknown[]> = {};
122 134 const phaseResults: Record<string, PhaseSlotResult[]> = {};

extensions/ultra/index.ts

Modified
9 9 // overlay, the `onUpdate` tool surface — is exactly the part only the Task-10
10 10 // live gate can prove; the harness test asserts registration only.
12 -import { randomUUID } from "node:crypto";
13 12 import { readFileSync } from "node:fs";
14 13 import { complete } from "@earendil-works/pi-ai/compat";
15 14 import {
16 15 createAgentSession,
17 16 DefaultResourceLoader,
18 17 type ExtensionAPI,
19 - type ExtensionCommandContext,
20 18 type ExtensionContext,
21 19 getAgentDir,
22 20 getMarkdownTheme,
25 23 } from "@earendil-works/pi-coding-agent";
26 24 import { Markdown } from "@earendil-works/pi-tui";
27 25 import { Type } from "typebox";
26 +import { createActionSummarizer } from "./action-summarizer.ts";
28 27 import {
29 28 type CompleteApi,
30 29 makeCompleteFn,
35 34 type SessionFactoryDeps,
36 35 } from "./drivers.ts";
37 36 import { runWorkflow, type WorkflowResult } from "./engine.ts";
38 -import {
39 - createSessionWorkflowJournal,
40 - deleteWorkflowRun,
41 - listWorkflowRuns,
42 - saveWorkflowRun,
43 -} from "./journal.ts";
37 +import { createSessionWorkflowJournal } from "./journal.ts";
44 38 import { initialReducerState } from "./progress.ts";
45 39 import {
46 40 type BaseStepOptions,
53 47 import {
54 48 type DiscoveredWorkflow,
55 49 discoverWorkflows,
50 + missingWorkflowTools,
56 51 type WorkflowSpec,
57 52 WorkflowSpecSchema,
58 53 } from "./spec.ts";
59 54 import { renderUltraResult, UltraOverlayComponent } from "./ui.ts";
60 55 import {
61 - completeWorkflowNames,
62 - formatBackgroundAcceptedMessage,
63 - formatBackgroundResultMessage,
56 + completeUltraCommand,
64 57 formatUltraResultMarkdown,
65 58 formatWorkflowResultMessage,
66 59 makeToolProgressSink,
67 60 parseCommandLine,
61 + parseUltraCommand,
68 62 resolveWorkflowInput,
69 63 } from "./wire.ts";
129 + const unavailableToolError = (
130 + spec: WorkflowSpec,
131 + cfg: UltraSettings,
132 + ): string | undefined => {
133 + const available = pi
134 + .getAllTools()
135 + .filter(
136 + (tool) =>
137 + tool.name !== "workflow" &&
138 + (cfg.subagentExtensions === "all" ||
139 + tool.sourceInfo.source === "builtin"),
141 + .map((tool) => tool.name);
142 + const missing = missingWorkflowTools(spec, available);
143 + return missing.length > 0
144 + ? `ultra: workflow "${spec.name}" requires unavailable tools: ${missing.join(", ")}`
145 + : undefined;
146 + };
135 148 const createJournal = (
136 149 ctx: ExtensionContext,
137 150 spec: WorkflowSpec,
167 180 getAgentDir,
168 181 cwd: ctx.cwd,
169 182 loadExtensions: cfg.subagentExtensions === "all",
183 + loadSkills: cfg.subagentSkills === "all",
171 185 resolveModel: makeResolveModel({
172 186 modelRegistry: ctx.modelRegistry,
184 198 maxRetries: cfg.maxRetries,
185 199 schemas: spec.schemas,
186 200 modelTiers: cfg.modelTiers,
201 + actionSummarizer: createActionSummarizer(ctx, cfg.actionSummarizerModel),
188 203 return makeStepRunner(drivers, ctxLike, baseOpts);
191 - const activeRuns = new Map<string, AbortController>();
192 - const startBackground = (
193 - ctx: ExtensionContext,
194 - spec: WorkflowSpec,
195 - args: Record<string, unknown> | undefined,
196 - cfg: UltraSettings,
197 - runId: string = randomUUID(),
198 - ): string => {
199 - const controller = new AbortController();
200 - activeRuns.set(runId, controller);
201 - const startedAt = new Date().toISOString();
202 - saveWorkflowRun({
203 - id: runId,
204 - workflow: spec.name,
205 - cwd: ctx.cwd,
206 - startedAt,
207 - status: "running",
208 - spec,
209 - args,
210 - });
211 - void (async () => {
212 - try {
213 - const result = await runWorkflow(spec, args, {
214 - stepRunner: buildStepRunner(ctx, spec, cfg),
215 - concurrency: cfg.concurrency,
216 - journal: createJournal(ctx, spec, args),
217 - signal: controller.signal,
218 - });
219 - if (controller.signal.aborted) throw new Error("stopped");
220 - saveWorkflowRun({
221 - id: runId,
222 - workflow: spec.name,
223 - cwd: ctx.cwd,
224 - startedAt,
225 - status: "completed",
226 - spec,
227 - args,
228 - result,
229 - });
230 - pi.sendMessage({
231 - customType: "ultra-result",
232 - content: formatBackgroundResultMessage(runId, result),
233 - display: true,
234 - details: result,
235 - });
236 - } catch (err) {
237 - const message = err instanceof Error ? err.message : String(err);
238 - saveWorkflowRun({
239 - id: runId,
240 - workflow: spec.name,
241 - cwd: ctx.cwd,
242 - startedAt,
243 - status: "failed",
244 - spec,
245 - args,
246 - error: message,
247 - });
248 - pi.sendMessage({
249 - customType: "ultra-result",
250 - content: `ultra background run "${spec.name}" failed: ${runId}\n${message}`,
251 - display: true,
252 - details: {
253 - workflow: spec.name,
254 - phases: [],
255 - phaseResults: {},
256 - phaseFailures: {},
257 - result: { error: message },
258 - steered: false,
259 - tokenUsage: {
260 - input: 0,
261 - output: 0,
262 - total: 0,
263 - cacheRead: 0,
264 - cacheWrite: 0,
265 - cost: 0,
266 - },
267 - },
268 - });
269 - } finally {
270 - activeRuns.delete(runId);
271 - }
272 - })();
273 - return runId;
274 - };
276 206 // ── `workflow` tool (model-callable, mid-turn, read-only onUpdate surface) ──
277 207 pi.registerTool({
278 208 name: "workflow",
280 210 description:
281 211 "Run a declarative ultra multi-agent workflow: fan out sub-agents across phases under a " +
282 212 "concurrency cap and return aggregated JSON. Do not use it for an ordinary single-agent task; " +
283 - "use one agent only when isolation, background execution, an alternate model, resumability, or " +
284 - "strict structured output is specifically needed. Pass either a saved workflow `name` " +
213 + "use one agent only when isolation, an alternate model, resumability, or strict structured output " +
214 + "is specifically needed. Pass either a saved workflow `name` " +
285 215 "(+ optional `args`), or an inline `spec` object for a one-off run. For inline specs, " +
286 216 "schema strings must exactly name top-level schemas; interpolation uses only the closed grammar " +
287 217 "described on the spec fields.",
298 228 "Workflow input values exposed to prompts and selectors as {args.NAME}.",
301 - background: Type.Optional(
302 - Type.Boolean({
303 - description: "Start the workflow in the background and report later.",
304 - }),
305 - ),
307 232 renderResult: renderUltraResult,
308 233 async execute(_toolCallId, params, signal, onUpdate, ctx) {
321 246 const { spec, args } = resolveWorkflowInput(params, discover(ctx.cwd));
247 + const toolError = unavailableToolError(spec, cfg);
248 + if (toolError) throw new Error(toolError);
322 249 const stepRunner = buildStepRunner(ctx, spec, cfg);
323 - if ((params as { background?: boolean }).background === true) {
324 - const runId = startBackground(ctx, spec, args, cfg);
325 - return {
326 - content: [
327 - {
328 - type: "text",
329 - text: formatBackgroundAcceptedMessage(runId, spec.name),
330 - },
331 - ],
332 - details: initialReducerState({
333 - workflowName: spec.name,
334 - workflowDescription: spec.description,
335 - }),
336 - terminate: false,
337 - };
338 - }
339 250 // onProgress folds into the reducer + pushes through `onUpdate`; headless
340 251 // (no `onUpdate`) degrades to a no-op sink that still tracks final state.
341 252 const progress = makeToolProgressSink(onUpdate ?? (() => {}), {
360 271 pi.on("session_start", () => deactivateWorkflowTool());
362 - type SavedRun = ReturnType<typeof listWorkflowRuns>[number];
364 - const runLabel = (run: SavedRun): string =>
365 - `${run.status} · ${run.workflow} · ${run.id}`;
367 - const runActions = (run: SavedRun, cwd: string): string[] => {
368 - const active = activeRuns.has(run.id);
369 - return [
370 - "Inspect",
371 - active ? "Stop" : "Delete",
372 - ...(!active && run.spec && run.cwd === cwd ? ["Resume"] : []),
373 - ];
374 - };
376 - const runFallbackDetails = (run: SavedRun): WorkflowResult => ({
377 - workflow: run.workflow,
378 - phases: [],
379 - phaseResults: {},
380 - phaseFailures: {},
381 - result: { status: run.status, error: run.error },
382 - steered: false,
383 - tokenUsage: {
384 - input: 0,
385 - output: 0,
386 - total: 0,
387 - cacheRead: 0,
388 - cacheWrite: 0,
389 - cost: 0,
390 - },
391 - });
393 - const inspectRun = (run: SavedRun) => {
394 - pi.sendMessage({
395 - customType: "ultra-result",
396 - content: `ultra saved run ${run.id}\n\n${JSON.stringify(run, null, 2)}`,
397 - display: true,
398 - details:
399 - (run.result as WorkflowResult | undefined) ?? runFallbackDetails(run),
400 - });
401 - };
403 - const runAction = (
404 - run: SavedRun,
405 - action: string | undefined,
406 - ctx: ExtensionCommandContext,
407 - ) => {
408 - if (action === "Stop") {
409 - activeRuns.get(run.id)?.abort();
410 - ctx.ui.notify(`Stopping ultra run ${run.id}.`, "warning");
411 - return;
412 - }
413 - if (action === "Resume" && run.spec) {
414 - startBackground(ctx, run.spec, run.args, settings(ctx.cwd), run.id);
415 - ctx.ui.notify(`Resumed ultra run ${run.id}.`, "info");
416 - return;
417 - }
418 - if (action === "Delete") {
419 - activeRuns.get(run.id)?.abort();
420 - deleteWorkflowRun(undefined, run.id);
421 - ctx.ui.notify(`Deleted ultra run ${run.id}.`, "info");
422 - return;
423 - }
424 - if (action === "Inspect") inspectRun(run);
425 - };
427 - const browseRuns = async (ctx: ExtensionCommandContext) => {
428 - const runs = listWorkflowRuns();
429 - if (runs.length === 0) {
430 - ctx.ui.notify("No saved ultra runs.", "info");
431 - return;
432 - }
433 - const labels = runs.map(runLabel);
434 - const selected = await ctx.ui.select("Ultra runs", labels);
435 - const run = runs[labels.indexOf(selected ?? "")];
436 - if (!run) return;
437 - const action = await ctx.ui.select(
438 - `${run.workflow} · ${run.id}`,
439 - runActions(run, ctx.cwd),
440 - );
441 - runAction(run, action, ctx);
442 - };
444 273 const injectAuthoringPrompt = (request: string) => {
445 274 activateWorkflowTool();
446 275 pi.sendMessage(
457 - // ── `/ultra [runs | <name> [args] | <instructions>]` command ──
286 + // ── `/ultra [exec | run]` command ──
458 287 pi.registerCommand("ultra", {
459 288 description:
460 - "Run ultra workflows, browse saved runs, or author a workflow from instructions.",
461 - getArgumentCompletions: (prefix) => {
462 - const p = prefix.toLowerCase();
463 - const runs = "runs".startsWith(p)
464 - ? [{ value: "runs", label: "runs", description: "saved runs" }]
465 - : [];
466 - const workflows =
467 - completeWorkflowNames(discover(process.cwd()), prefix) ?? [];
468 - const items = [...runs, ...workflows];
469 - return items.length > 0 ? items : null;
470 - },
289 + "Enable ultra, execute workflow instructions, or run a saved workflow.",
290 + getArgumentCompletions: (prefix) =>
291 + completeUltraCommand(discover(process.cwd()), prefix),
471 292 handler: async (argString, ctx) => {
472 293 const cfg = settings(ctx.cwd);
473 294 if (!cfg.enabled) {
483 304 if (!ctx.hasUI) {
484 305 return;
486 - activateWorkflowTool();
488 - const trimmed = argString.trim();
489 - const discovered = discover(ctx.cwd);
490 - const parsed = parseCommandLine(trimmed);
491 - if (parsed.name === "runs") {
492 - await browseRuns(ctx);
307 + const command = parseUltraCommand(argString);
308 + if (command.subcommand === "activate") {
309 + activateWorkflowTool();
310 + ctx.ui.notify("ultra: workflow tool enabled.", "info");
493 311 return;
495 - if (!parsed.name) {
313 + if (command.subcommand === "exec") {
314 + injectAuthoringPrompt(command.instructions);
315 + ctx.ui.notify("ultra authoring prompt sent.", "info");
316 + return;
318 + if (command.subcommand === "invalid") {
496 319 ctx.ui.notify(
497 - "Usage: /ultra runs | /ultra <workflow> [args] | /ultra <instructions>",
320 + "Usage: /ultra | /ultra exec <instructions> | /ultra run <workflow> [args]",
498 321 "warning",
500 323 return;
502 - if (!discovered.some((workflow) => workflow.name === parsed.name)) {
503 - injectAuthoringPrompt(trimmed);
504 - ctx.ui.notify("ultra authoring prompt sent.", "info");
505 - return;
506 - }
326 + const discovered = discover(ctx.cwd);
327 + const parsed = parseCommandLine(command.invocation);
508 328 let resolved: { spec: WorkflowSpec; args?: Record<string, unknown> };
510 330 resolved = resolveWorkflowInput(
520 340 return;
522 342 const { spec, args: runArgs } = resolved;
343 + const toolError = unavailableToolError(spec, cfg);
344 + if (toolError) {
345 + ctx.ui.notify(toolError, "error");
346 + return;
523 348 const stepRunner = buildStepRunner(ctx, spec, cfg);
525 350 // The command owns its AbortController (idle ⇒ ctx.signal is undefined).
526 351 // Global Esc / per-agent abort fire it; the engine fans session.abort().
527 352 const controller = new AbortController();
528 353 const result = await ctx.ui.custom<WorkflowResult | null>(
529 - (tui, theme, _kb, done) => {
354 + (tui, theme, keybindings, done) => {
530 355 const overlay = new UltraOverlayComponent({
531 356 host: tui,
532 357 theme,
534 359 workflowName: spec.name,
535 360 workflowDescription: spec.description,
536 361 toolUi: tui,
362 + toolsExpanded: () => ctx.ui.getToolsExpanded(),
363 + hideToolPreviews: cfg.actionSummarizerModel !== null,
364 + isToolsExpandKey: (data) =>
365 + keybindings.matches(data, "app.tools.expand"),
366 + onToolsExpand: () =>
367 + ctx.ui.setToolsExpanded(!ctx.ui.getToolsExpanded()),
537 368 cwd: ctx.cwd,
539 370 // P3: return the Component SYNCHRONOUSLY; run the engine DETACHED, or
555 386 return overlay;
389 + overlay: true,
390 + overlayOptions: {
391 + anchor: "bottom-left",
392 + width: "100%",
393 + maxHeight: "100%",
394 + },
395 + },
559 398 if (!result) {

extensions/ultra/journal.ts

Modified
1 1 import { createHash } from "node:crypto";
2 -import fs from "node:fs";
3 -import os from "node:os";
4 -import path from "node:path";
5 -import { CONFIG_DIR_NAME, getAgentDir } from "@earendil-works/pi-coding-agent";
6 2 import type { WorkflowJournal } from "./engine.ts";
7 3 import type { RunStepResult } from "./runner.ts";
8 4 import type { WorkflowSpec } from "./spec.ts";
31 27 appendEntry: (customType: string, data: unknown) => void;
34 -export type WorkflowRunStatus = "running" | "completed" | "failed";
36 -export interface SavedWorkflowRun {
37 - id: string;
38 - workflow: string;
39 - cwd: string;
40 - startedAt: string;
41 - status: WorkflowRunStatus;
42 - spec?: WorkflowSpec;
43 - args?: Record<string, unknown>;
44 - result?: unknown;
45 - error?: string;
48 -export interface SaveWorkflowRunOptions extends SavedWorkflowRun {
49 - home?: string;
52 30 function stableJson(value: unknown): string {
53 31 if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`;
54 32 if (value && typeof value === "object") {
74 52 .slice(0, 32);
77 -function agentDir(home = os.homedir()): string {
78 - return home === os.homedir()
79 - ? getAgentDir()
80 - : path.join(home, CONFIG_DIR_NAME, "agent");
83 -function runsDir(home = os.homedir()): string {
84 - return path.join(agentDir(home), "ultra-runs");
87 -function safeRunId(id: string): boolean {
88 - return /^[A-Za-z0-9_-]+$/.test(id);
91 -export function saveWorkflowRun(opts: SaveWorkflowRunOptions): void {
92 - if (!safeRunId(opts.id)) throw new Error("Invalid ultra run id");
93 - const { home, ...run } = opts;
94 - const file = path.join(runsDir(home), `${run.id}.json`);
95 - fs.mkdirSync(path.dirname(file), { recursive: true });
96 - fs.writeFileSync(file, `${JSON.stringify(run, null, 2)}\n`, "utf8");
99 -export function listWorkflowRuns(home?: string): SavedWorkflowRun[] {
100 - try {
101 - return fs
102 - .readdirSync(runsDir(home))
103 - .filter((file) => file.endsWith(".json"))
104 - .flatMap((file) => {
105 - try {
106 - return [
107 - JSON.parse(
108 - fs.readFileSync(path.join(runsDir(home), file), "utf8"),
109 - ) as SavedWorkflowRun,
110 - ];
111 - } catch {
112 - return [];
113 - }
114 - })
115 - .sort((a, b) => b.startedAt.localeCompare(a.startedAt));
116 - } catch {
117 - return [];
118 - }
121 -export function deleteWorkflowRun(
122 - home: string | undefined,
123 - id: string,
124 -): boolean {
125 - if (!safeRunId(id)) return false;
126 - try {
127 - fs.unlinkSync(path.join(runsDir(home), `${id}.json`));
128 - return true;
129 - } catch {
130 - return false;
131 - }
134 55 function parseJournalEntry(value: unknown): JournalEntryData | undefined {
135 56 if (!value || typeof value !== "object") return undefined;
136 57 const data = value as Partial<JournalEntryData>;

extensions/ultra/progress.ts

Modified
53 53 /** Raw tool result for native tool rendering. */
54 54 result?: unknown;
55 55 isError?: boolean;
56 + isPartial?: boolean;
56 57 /** Short resolved task summary shown on progress rows. */
57 58 summary?: string;
58 59 /** Exact resolved sub-agent prompt, shown as the first overlay transcript message. */
59 60 prompt?: string;
61 + /** LLM-authored board phrase for an action; takes precedence over raw args. */
62 + actionSummary?: string;
60 63 /** Streamed delta text, a tool action/result preview, or a coarse status line. */
61 64 text?: string;
62 65 /** Terminal/error marker: `end` carries the run outcome; `action` may carry `"error"`. */
115 118 return typeof value === "object" && value !== null && !Array.isArray(value);
121 +/** Short preview of arbitrary JSON without dumping nested bodies. */
122 +function previewValue(value: unknown): string {
123 + if (value === undefined) return "";
124 + if (value === null) return "null";
125 + if (typeof value === "string") return cap(firstLine(value) ?? "");
126 + if (typeof value !== "object") return String(value);
127 + if (Array.isArray(value)) return plural(value.length, "item");
128 + const entries = Object.entries(value);
129 + if (entries.length === 0) return "{}";
130 + const shown = entries.slice(0, 4).map(([key, entry]) => {
131 + if (Array.isArray(entry)) return `${key}=[${entry.length}]`;
132 + if (isRecord(entry)) return `${key}={${Object.keys(entry).length}}`;
133 + return `${key}=${cap(stringify(entry))}`;
134 + });
135 + if (entries.length > shown.length)
136 + shown.push(`+${entries.length - shown.length} more`);
137 + return cap(shown.join(" · "));
118 140 /** Short preview of a tool call's args for the board/transcript. */
119 141 function previewArgs(args: unknown): string {
120 142 if (args === undefined || args === null) return "";
121 - return cap(stringify(args));
143 + return previewValue(args);
124 146 function firstLine(value: unknown): string | undefined {
125 147 return typeof value === "string" ? value.split("\n")[0]?.trim() : undefined;
128 -function conciseArgs(toolName: string | undefined, args: unknown): string {
129 - if (!toolName || !isRecord(args)) return previewArgs(args);
150 +function pathArg(args: Record<string, unknown>): string | undefined {
151 + return firstLine(args.path) ?? firstLine(args.file);
154 +function searchArgs(args: Record<string, unknown>): string {
155 + const pattern = firstLine(args.pattern) ?? firstLine(args.query);
156 + const path = pathArg(args);
157 + if (pattern && path) return `${pattern} in ${path}`;
158 + return pattern ?? path ?? previewArgs(args);
161 +function actionIntent(toolName: string, args: unknown): string {
162 + if (!isRecord(args)) return `Using ${toolName}`;
163 + const path = pathArg(args);
130 164 switch (toolName) {
131 165 case "bash": {
132 166 const command = firstLine(args.command);
133 - return command ? `$ ${cap(command)}` : previewArgs(args);
167 + return command ? `Running ${cap(command)}` : "Running command";
135 169 case "read":
170 + return path ? `Reading ${path}` : "Reading input";
171 + case "write":
172 + return path ? `Writing ${path}` : "Writing file";
173 + case "edit":
174 + return path ? `Editing ${path}` : "Editing file";
136 175 case "ls":
137 - return firstLine(args.path) ?? previewArgs(args);
138 - case "grep": {
139 - const pattern = firstLine(args.pattern);
140 - const path = firstLine(args.path);
141 - if (pattern && path) return `${pattern} in ${path}`;
142 - return pattern ?? path ?? previewArgs(args);
143 - }
144 - case "find": {
145 - const pattern = firstLine(args.pattern);
146 - const path = firstLine(args.path);
147 - if (pattern && path) return `${pattern} in ${path}`;
148 - return pattern ?? path ?? previewArgs(args);
149 - }
176 + return path ? `Listing ${path}` : "Listing files";
177 + case "grep":
178 + return `Searching ${searchArgs(args)}`;
179 + case "find":
180 + return `Finding ${searchArgs(args)}`;
181 + case "web_fetch":
182 + return `Fetching ${firstLine(args.url) ?? "URL"}`;
183 + case "web_search":
184 + return `Searching web for ${firstLine(args.query) ?? "query"}`;
185 + case "understand_image":
186 + return `Inspecting ${firstLine(args.image_source) ?? "image"}`;
150 187 case "structured_output":
151 188 return structuredOutputSummary(args);
152 - default:
153 - return previewArgs(args);
189 + default: {
190 + const detail = previewArgs(args);
191 + return detail ? `Using ${toolName} · ${detail}` : `Using ${toolName}`;
174 213 * structured-output payload path (P1) — falling back to `result.content`, then
175 214 * the whole result.
177 -function previewResult(result: unknown): string {
178 - if (result === undefined || result === null) return "";
179 - if (typeof result === "object") {
180 - const r = result as { details?: unknown; content?: unknown };
181 - if (r.details !== undefined && r.details !== null)
182 - return cap(stringify(r.details));
183 - if (r.content !== undefined && r.content !== null)
184 - return cap(stringify(r.content));
216 +function contentText(content: unknown): string | undefined {
217 + if (typeof content === "string") return firstLine(content);
218 + if (!Array.isArray(content)) return undefined;
219 + for (const part of content) {
220 + if (!isRecord(part) || part.type !== "text") continue;
221 + const text = firstLine(part.text);
222 + if (text) return text;
186 - return cap(stringify(result));
224 + return undefined;
227 +function previewResult(result: unknown): string {
228 + if (result === undefined) return "";
229 + if (result === null) return "null";
230 + if (isRecord(result)) {
231 + if (result.details !== undefined && result.details !== null)
232 + return previewValue(result.details);
233 + const text = contentText(result.content);
234 + if (text) return cap(text);
235 + if (result.content !== undefined) return previewValue(result.content);
237 + return previewValue(result);
214 265 text: previewArgs(event.args),
268 + case "tool_execution_update":
269 + return {
270 + ...base,
271 + kind: "action",
272 + toolName: event.toolName,
273 + toolCallId: event.toolCallId,
274 + args: event.args,
275 + result: event.partialResult,
276 + isPartial: true,
277 + text: previewResult(event.partialResult),
278 + };
217 280 case "tool_execution_end":
218 281 return {
219 282 ...base,
222 285 toolCallId: event.toolCallId,
223 286 result: event.result,
224 287 isError: event.isError,
288 + isPartial: false,
225 289 text: previewResult(event.result),
226 290 status: event.isError ? "error" : undefined,
288 352 args?: unknown;
289 353 result?: unknown;
290 354 isError?: boolean;
355 + isPartial?: boolean;
291 356 fallback: string;
349 414 /** The coarse board label for an `action` event. */
350 415 function actionLabel(event: AgentProgressEvent): string {
351 - const text =
352 - event.args !== undefined
353 - ? conciseArgs(event.toolName, event.args)
354 - : (event.text ?? "");
355 - if (event.toolName === "structured_output")
356 - return text ? `structured output · ${text}` : "structured output";
357 - if (event.toolName)
358 - return text ? `${event.toolName} ${text}` : event.toolName;
359 - return text;
416 + if (event.actionSummary) return event.actionSummary;
417 + if (event.result !== undefined) {
418 + const result = event.text ?? previewResult(event.result);
419 + if (event.isError) return result ? `Failed · ${result}` : "Failed";
420 + return result ? `Result · ${result}` : "Finished";
422 + if (event.toolName === "structured_output") {
423 + const detail = isRecord(event.args)
424 + ? structuredOutputSummary(event.args)
425 + : event.text;
426 + return detail ? `structured output · ${detail}` : "structured output";
428 + if (event.toolName && event.args !== undefined)
429 + return actionIntent(event.toolName, event.args);
430 + if (event.toolName && event.text) return `${event.toolName} ${event.text}`;
431 + if (event.toolName) return `Using ${event.toolName}`;
432 + return event.text ?? "";
362 435 /** Text contributed to the per-agent transcript by this event (may be empty). */
386 459 args: event.args,
387 460 result: event.result,
388 461 isError: event.isError,
462 + isPartial: event.isPartial,
389 463 fallback,
562 636 : state.transcripts;
564 638 const item = transcriptItem(event);
565 - const transcriptItems = item
566 - ? {
567 - ...state.transcriptItems,
568 - [event.agentId]: [
569 - ...(state.transcriptItems[event.agentId] ?? []),
570 - item,
571 - ],
572 - }
573 - : state.transcriptItems;
639 + let transcriptItems = state.transcriptItems;
640 + if (item) {
641 + const current = state.transcriptItems[event.agentId] ?? [];
642 + const toolIndex =
643 + item.kind === "tool" && item.toolCallId
644 + ? current.findIndex(
645 + (entry) =>
646 + entry.kind === "tool" && entry.toolCallId === item.toolCallId,
648 + : -1;
649 + const next =
650 + toolIndex < 0
651 + ? [...current, item]
652 + : current.map((entry, index) =>
653 + index === toolIndex && entry.kind === "tool" && item.kind === "tool"
654 + ? {
655 + ...entry,
656 + args: item.args ?? entry.args,
657 + result: item.result ?? entry.result,
658 + isError: item.isError ?? entry.isError,
659 + isPartial: item.isPartial ?? entry.isPartial,
660 + fallback: item.fallback || entry.fallback,
662 + : entry,
663 + );
664 + transcriptItems = {
665 + ...state.transcriptItems,
666 + [event.agentId]: next,
667 + };
575 670 const phases = state.phases.some((phase) => phase.phase === event.phase)
576 671 ? state.phases

extensions/ultra/prompts/authoring-workflows.md

Modified
1 1 # Authoring ultra workflows
3 3 These instructions help you co-design a **persisted** ultra workflow with the
4 -user and save it to disk so they can re-run it as `/ultra <name>`. The canonical
4 +user and save it to disk so they can re-run it as `/ultra run <name>`. The canonical
5 5 spec format is injected below — use it before writing any spec; do not restate it
6 6 from memory.
27 27 ## Single-agent gate
29 29 Do not create or invoke ultra for an ordinary task that one agent can perform directly.
30 -A workflow with one agent is justified only when it specifically needs context isolation, background execution, an alternate model, resumability, strict structured output, or future reuse as orchestration.
30 +A workflow with one agent is justified only when it specifically needs context isolation, an alternate model, resumability, strict structured output, or future reuse as orchestration.
31 31 Otherwise perform the task directly and skip ultra's orchestration overhead.
33 33 ## Authoring flow (persisted)
48 48 - `summary` is one concise progress-UI sentence, uses the same interpolation grammar as `prompt`, and must describe the concrete sub-agent task without paraphrasing its result.
50 50 3. **Apply the read-only fan-out rule.** A `fanout` step with tools must stay
51 - within `read`, `grep`, `find`, `ls`, `bash` **and** the prompt must forbid
52 - mutation — parallel writers race. Put any writing work in a `single` phase.
51 + within `read`, `grep`, `find`, `ls`, `bash`, `web_search`, `web_fetch`,
52 + `web_map` **and** the prompt must forbid mutation — parallel writers race.
53 + Every listed tool must be installed and available or Ultra refuses to start the workflow.
54 + Put any writing work in a `single` phase.
54 56 4. **Design schemas and join-keys.** Give a step a `schema` (named in `schemas`,
55 57 or inline) when a later phase consumes its output. When phase B fans out over
78 80 surfaces a TypeBox error naming the offending path if anything is malformed —
79 81 fix and re-run until it loads clean. One caveat: a JSON _syntax_ error (a
80 82 trailing comma, a missing quote) makes the file invisible to discovery
81 - entirely, so `/ultra <name>` reports `unknown workflow` rather than a schema
83 + entirely, so `/ultra run <name>` reports `unknown workflow` rather than a schema
82 84 error — if a saved workflow seems to vanish, confirm it is valid JSON first.
84 86 8. **Save it.** Write the spec as `<name>.json` (filename matching the spec's
89 91 Ask which scope only if it isn't clear from context (a repo-specific review
90 92 workflow → project; a personal utility → global).
92 -9. **Offer a test run.** After saving, offer to run it: `/ultra <name>` (with
94 +9. **Offer a test run.** After saving, offer to run it: `/ultra run <name>` (with
93 95 any `args` the spec's `hint` declares). Tighten prompts/schemas from what the
94 96 first run surfaces.

extensions/ultra/prompts/references/spec-format.md

Modified
10 10 | Field | Required | Type | Meaning |
11 11 | ------------- | -------- | --------------------------- | -------------------------------------------------------------- |
12 -| `name` | yes | string (non-empty) | Workflow id. The file is invoked as `/ultra <name>`. |
12 +| `name` | yes | string (non-empty) | Workflow id. The file is invoked as `/ultra run <name>`. |
13 13 | `description` | no | string | One line shown in the picker. |
14 14 | `args` | no | object (`{ hint?, ... }`) | Declares run inputs; `hint` is the usage shown after the name. |
15 15 | `schemas` | no | record<string, JSON Schema> | Named, reusable output schemas referenced by steps. |
17 17 | `return` | no | string (one selector token) | Selects the workflow's final value from phase results. |
19 19 `args` may also declare `params`: an ordered list of input names. A bare
20 -`/ultra <name> tokA tokB` line maps tokens onto them positionally; the last declared parameter absorbs all remaining tokens so it can carry free text. Values are exposed as `{args.NAME}`. Without `params`, pass inputs as a trailing JSON object — `/ultra <name> {"k": "v"}`. A missing `{args.NAME}` renders to the empty string.
20 +`/ultra run <name> tokA tokB` maps tokens onto them positionally; the last declared parameter absorbs all remaining tokens so it can carry free text. Values are exposed as `{args.NAME}`. Without `params`, pass inputs as a trailing JSON object: `/ultra run <name> {"k": "v"}`. A missing `{args.NAME}` renders to the empty string.
22 22 ## Phases
43 43 ### The read-only fan-out rule (v1)
45 45 A `fanout` step that grants tools must stay within the read-only allowlist —
46 -`read`, `grep`, `find`, `ls`, `bash` — and the prompt must instruct the agent
47 -not to mutate anything. Parallel agents that write race each other; v1 keeps
48 -fan-out side-effect-free. A `single` synthesis phase can be given broader tools
49 -if a workflow genuinely needs to write.
46 +`read`, `grep`, `find`, `ls`, `bash`, `web_search`, `web_fetch`, `web_map` — and
47 +the prompt must instruct the agent not to mutate anything. Parallel agents that
48 +write race each other; v1 keeps fan-out side-effect-free. A `single` synthesis
49 +phase can be given broader tools if a workflow genuinely needs to write.
50 +Every listed tool must be available to sub-agents or Ultra refuses to start the workflow.
51 52 ## Steps
195 196 Fan out a review across dimensions, then fan out verification over the flattened
196 197 findings. `Findings.file` → `Verdict.file` is the join key; `return` filters to
197 198 the verdicts the verifier judged `real`. The single `params` entry lets a bare
198 -`/ultra review main` map `main` to `{args.base}` (a missing arg renders empty, so
199 +`/ultra run review main` maps `main` to `{args.base}` (a missing arg renders empty, so
199 200 the diff falls back to the working tree).
201 202 ```json

extensions/ultra/runner.ts

Modified
32 32 import type { ProviderHeaders } from "@earendil-works/pi-ai";
33 33 import type { AgentSessionEvent } from "@earendil-works/pi-coding-agent";
34 +import type { ActionSummarizer } from "./action-summarizer.ts";
34 35 import {
35 36 type AgentControls,
36 37 type AgentProgressEvent,
185 186 modelTiers?: ModelTiers;
186 187 /** Optional extra system-prompt text for the tool-less path. */
187 188 systemPrompt?: string;
189 + /** Optional best-effort LLM summary for tool actions shown on the board. */
190 + actionSummarizer?: ActionSummarizer;
190 193 /** Full options for one `runStep` call (run-level ⊕ per-call). */
207 210 onProgress?: (event: UltraProgressEvent) => void;
210 -/** The single function shape the engine (Task 6) consumes. */
211 -export type StepRunner = (args: StepRunnerArgs) => Promise<RunStepResult>;
213 +/** The workflow-scoped runner consumed by the engine. */
214 +export type StepRunner = ((args: StepRunnerArgs) => Promise<RunStepResult>) & {
215 + dispose?(): void;
213 218 /** Name of the injected terminating StructuredOutput tool (appended to the allowlist). */
214 219 export const STRUCTURED_OUTPUT_TOOL_NAME = "structured_output";
761 766 customTools: [structuredOutputTool],
762 767 sessionManager: ctxLike.newSessionManager(),
764 - unsubscribe = observeToolSession(session, opts, meta, state);
769 + unsubscribe = observeToolSession(session, opts, meta, state, step.summary);
765 770 return await promptToolSession(
766 771 session,
767 772 step.prompt,
828 833 opts: RunStepOptions,
829 834 meta: ToolStepMeta,
830 835 state: ToolStepState,
836 + taskSummary: string,
831 837 ): () => void {
832 838 const controls = toolControls(session, state);
833 - return session.subscribe((event) => {
839 + let active = true;
840 + let actionVersion = 0;
841 + const unsubscribe = session.subscribe((event) => {
834 842 if (event.type === "message_start" && event.message.role === "assistant")
835 843 state.transcript = "";
836 844 if (event.type === "agent_end")
841 849 event.type === "message_update" ? event.assistantMessageEvent : undefined;
842 850 if (update?.type === "text_delta")
843 851 state.transcript = appendTranscriptTail(state.transcript, update.delta);
852 + if (
853 + (event.type === "tool_execution_start" ||
854 + event.type === "tool_execution_update" ||
855 + event.type === "tool_execution_end") &&
856 + event.toolName === STRUCTURED_OUTPUT_TOOL_NAME
858 + return;
844 859 const mapped = mapSessionEvent(event, opts.agentId, opts.phase);
845 860 if (!mapped) return;
861 + const toolStart = event.type === "tool_execution_start" ? event : undefined;
862 + const summarizeAction = toolStart ? opts.actionSummarizer : undefined;
846 863 opts.onProgress?.({
847 864 ...mapped,
848 865 ...meta,
849 866 usage: mapped.kind === "end" ? state.usage : undefined,
850 867 controls,
869 + if (!summarizeAction || !toolStart) return;
870 + const version = ++actionVersion;
871 + void summarizeAction({
872 + agentId: opts.agentId,
873 + toolName: toolStart.toolName,
874 + args: toolStart.args,
875 + taskSummary,
876 + signal: opts.signal,
877 + })
878 + .then((summary) => {
879 + if (!active || version !== actionVersion || !summary) return;
880 + opts.onProgress?.({
881 + agentId: opts.agentId,
882 + phase: opts.phase,
883 + kind: "action",
884 + toolName: toolStart.toolName,
885 + toolCallId: toolStart.toolCallId,
886 + actionSummary: summary,
887 + ...meta,
888 + controls,
889 + });
890 + })
891 + .catch(() => {});
893 + return () => {
894 + if (state.manuallyAborted || opts.signal?.aborted) {
895 + active = false;
896 + opts.actionSummarizer?.cancel?.(opts.agentId);
897 + } else {
898 + opts.actionSummarizer?.finish?.(opts.agentId);
900 + unsubscribe();
901 + };
855 904 function toolControls(
917 966 ctxLike: RunnerContext,
918 967 baseOpts: BaseStepOptions,
919 968 ): StepRunner {
920 - return (args: StepRunnerArgs) =>
969 + const runner = (args: StepRunnerArgs) =>
921 970 runStep(args.step, ctxLike, drivers, {
922 971 ...baseOpts,
923 972 item: args.item,
926 975 signal: args.signal,
927 976 onProgress: args.onProgress,
978 + runner.dispose = () => baseOpts.actionSummarizer?.dispose?.();
979 + return runner;

extensions/ultra/settings.ts

Modified
17 17 * (`"all"`, default) or keep sub-agents isolated to core tools (`"none"`).
18 18 * The `step.tools` allowlist still gates which tools are actually active. */
19 19 subagentExtensions: "none" | "all";
20 + /** Load host skills into sub-agents (`"all"`, default) or omit them (`"none"`). */
21 + subagentSkills: "none" | "all";
20 22 /** Optional local aliases used by step.model: small|medium|big. */
21 23 modelTiers: ModelTiers;
24 + /** Fast model used to turn raw tool args into board action phrases; null disables it. */
25 + actionSummarizerModel: string | null;
24 28 /** Documented defaults (design §Settings table). */
27 31 concurrency: 5,
28 32 maxRetries: 2,
29 33 subagentExtensions: "all",
34 + subagentSkills: "all",
30 35 modelTiers: {},
36 + actionSummarizerModel: null,
33 39 function isPlainObject(v: unknown): v is Record<string, unknown> {
60 66 if (typeof ultra.maxRetries === "number") out.maxRetries = ultra.maxRetries;
61 67 if (ultra.subagentExtensions === "none" || ultra.subagentExtensions === "all")
62 68 out.subagentExtensions = ultra.subagentExtensions;
69 + if (ultra.subagentSkills === "none" || ultra.subagentSkills === "all")
70 + out.subagentSkills = ultra.subagentSkills;
63 71 const modelTiers = pickModelTiers(ultra.modelTiers);
64 72 if (modelTiers) out.modelTiers = modelTiers;
73 + if (ultra.actionSummarizerModel === null) out.actionSummarizerModel = null;
74 + else if (
75 + typeof ultra.actionSummarizerModel === "string" &&
76 + ultra.actionSummarizerModel.trim()
77 + )
78 + out.actionSummarizerModel = ultra.actionSummarizerModel.trim();
65 79 return out;
81 95 p.subagentExtensions === "none" || p.subagentExtensions === "all"
82 96 ? p.subagentExtensions
83 97 : into.subagentExtensions,
98 + subagentSkills:
99 + p.subagentSkills === "none" || p.subagentSkills === "all"
100 + ? p.subagentSkills
101 + : into.subagentSkills,
84 102 modelTiers: isPlainObject(p.modelTiers)
85 103 ? { ...into.modelTiers, ...p.modelTiers }
86 104 : into.modelTiers,
105 + actionSummarizerModel:
106 + p.actionSummarizerModel === null ||
107 + typeof p.actionSummarizerModel === "string"
108 + ? p.actionSummarizerModel
109 + : into.actionSummarizerModel,
88 111 { ...ULTRA_DEFAULTS },

extensions/ultra/spec.ts

Modified
71 71 tools: Type.Optional(
72 72 Type.Array(Type.String(), {
73 73 description:
74 - "Sub-agent tool allowlist. Fanout may use only read, grep, find, ls, bash, and its prompt must forbid mutation. Put writing tools in a single phase.",
74 + "Sub-agent tool allowlist. Every listed tool must be available or Ultra refuses to start. Fanout may use only read, grep, find, ls, bash, web_search, web_fetch, and web_map, and its prompt must forbid mutation. Put writing tools in a single phase.",
77 77 model: Type.Optional(
309 309 spec: WorkflowSpec;
312 +export function missingWorkflowTools(
313 + spec: WorkflowSpec,
314 + availableTools: Iterable<string>,
315 +): string[] {
316 + const available = new Set(availableTools);
317 + return [...new Set(spec.phases.flatMap((phase) => phase.step.tools ?? []))]
318 + .filter((tool) => !available.has(tool))
319 + .sort();
312 322 export interface DiscoverOptions {
313 323 /** Override the home directory for global workflow discovery in tests. */
314 324 home?: string;

extensions/ultra/ui.ts

Modified
503 503 workflowDescription?: string;
504 504 /** Optional Pi TUI handle; when present, tool transcript rows use Pi's native tool renderer. */
505 505 toolUi?: TUI;
506 + toolsExpanded?: () => boolean;
507 + hideToolPreviews?: boolean;
508 + isToolsExpandKey?: (data: string) => boolean;
509 + onToolsExpand?: () => void;
506 510 cwd?: string;
557 561 private steerBuffer = "";
558 562 private steeredFlag = false;
559 563 private abortArmedAt: number | undefined;
564 + private readonly transcriptTools = new Map<
565 + string,
567 + component: ToolExecutionComponent;
568 + args: unknown;
569 + result: unknown;
570 + isError: boolean | undefined;
571 + isPartial: boolean | undefined;
572 + expanded: boolean;
574 + >();
561 576 private readonly host: OverlayHost;
562 577 private readonly theme: Theme;
564 579 private readonly now: () => number;
565 580 private readonly throttleMs: number;
566 581 private readonly toolUi: TUI | undefined;
582 + private readonly toolsExpanded: () => boolean;
583 + private readonly hideToolPreviews: boolean;
584 + private readonly isToolsExpandKey: (data: string) => boolean;
585 + private readonly onToolsExpand: () => void;
567 586 private readonly cwd: string | undefined;
569 588 constructor(options: UltraOverlayOptions) {
577 596 this.now = options.now ?? Date.now;
578 597 this.throttleMs = options.throttleMs ?? DEFAULT_THROTTLE_MS;
579 598 this.toolUi = options.toolUi;
599 + this.toolsExpanded = options.toolsExpanded ?? (() => false);
600 + this.hideToolPreviews = options.hideToolPreviews ?? false;
601 + this.isToolsExpandKey = options.isToolsExpandKey ?? (() => false);
602 + this.onToolsExpand = options.onToolsExpand ?? (() => {});
580 603 this.cwd = options.cwd;
619 642 // ---- input ------------------------------------------------------------
621 644 handleInput(data: string): void {
645 + if (this.isToolsExpandKey(data)) {
646 + this.onToolsExpand();
647 + this.host.requestRender();
648 + return;
622 650 if (this.steerMode) {
623 651 this.handleSteerInput(data);
624 652 return;
845 + private transcriptTool(
846 + agentId: string,
847 + item: TranscriptItem & { kind: "tool" },
848 + index: number,
849 + ): ToolExecutionComponent | undefined {
850 + if (!this.toolUi || !this.cwd) return undefined;
851 + const key = `${agentId}:${item.toolCallId ?? index}`;
852 + let cached = this.transcriptTools.get(key);
853 + if (!cached) {
854 + const component = new ToolExecutionComponent(
855 + item.toolName,
856 + item.toolCallId ?? key,
857 + item.args ?? {},
858 + undefined,
859 + undefined,
860 + this.toolUi,
861 + this.cwd,
862 + );
863 + component.markExecutionStarted();
864 + component.setArgsComplete();
865 + cached = {
866 + component,
867 + args: item.args,
868 + result: undefined,
869 + isError: undefined,
870 + isPartial: undefined,
871 + expanded: false,
872 + };
873 + this.transcriptTools.set(key, cached);
874 + } else if (item.args !== undefined && item.args !== cached.args) {
875 + cached.component.updateArgs(item.args);
876 + cached.args = item.args;
878 + if (
879 + item.result !== undefined &&
880 + (item.result !== cached.result ||
881 + item.isError !== cached.isError ||
882 + item.isPartial !== cached.isPartial)
883 + ) {
884 + cached.component.updateResult(
885 + normalizeToolResult(item),
886 + item.isPartial ?? false,
887 + );
888 + cached.result = item.result;
889 + cached.isError = item.isError;
890 + cached.isPartial = item.isPartial;
892 + const expanded = this.toolsExpanded();
893 + if (expanded !== cached.expanded) {
894 + cached.component.setExpanded(expanded);
895 + cached.expanded = expanded;
897 + return cached.component;
817 900 private transcriptBodyLines(
818 901 agentId: string | undefined,
819 902 width: number,
830 913 lines.push(...transcriptLines(text, width));
831 914 text = "";
833 - for (const item of items) {
916 + for (const [index, item] of items.entries()) {
834 917 if (item.kind === "text") {
835 918 text += item.text;
836 919 continue;
841 924 lines.push(...transcriptLines(item.text, width));
842 925 continue;
844 - if (this.toolUi && this.cwd) {
845 - const tool = new ToolExecutionComponent(
846 - item.toolName,
847 - item.toolCallId ?? `${agentId}-${lines.length}`,
848 - item.args ?? {},
849 - undefined,
850 - undefined,
851 - this.toolUi,
852 - this.cwd,
853 - );
854 - tool.markExecutionStarted();
855 - tool.setArgsComplete();
856 - if (item.result !== undefined)
857 - tool.updateResult(normalizeToolResult(item));
858 - lines.push(...tool.render(width));
859 - } else if (item.fallback) {
927 + if (this.hideToolPreviews && !this.toolsExpanded()) continue;
928 + const tool = this.transcriptTool(agentId, item, index);
929 + if (tool) lines.push(...tool.render(width));
930 + else if (item.fallback)
860 931 lines.push(...transcriptLines(`> ${item.fallback}`, width));
863 933 flushText();
864 934 return lines;
992 1062 this.theme.fg(
994 - fitLine("s steer · x abort agent · Esc Esc abort all", w),
1064 + fitLine(
1065 + `s steer · x abort agent · ${keyHint("app.tools.expand", "tools")} · Esc Esc abort all`,
1066 + w,
1067 + ),
1002 1075 invalidate(): void {
1003 - // No cached render state to drop.
1076 + for (const { component } of this.transcriptTools.values())
1077 + component.invalidate();

extensions/ultra/wire.ts

Modified
55 55 return value === undefined || value === null ? [] : [value];
58 +interface ReviewReport {
59 + analysis: string;
60 + findings: unknown[];
61 + recommendations: string[];
64 +interface ResearchReport {
65 + answerStatus: string;
66 + executiveSummary: string;
67 + analysis: string;
68 + findings: unknown[];
69 + contradictions: string[];
70 + gaps: string[];
71 + recommendations: string[];
72 + sources: unknown[];
75 +function stringArray(value: unknown): string[] {
76 + return Array.isArray(value)
77 + ? value.filter(
78 + (item): item is string => typeof item === "string" && item.length > 0,
79 + )
80 + : [];
83 +function reviewReport(result: WorkflowResult): ReviewReport | undefined {
84 + if (result.workflow !== "review") return undefined;
85 + const items = resultItems(result.result);
86 + if (items.length !== 1) return undefined;
87 + const value = record(items[0]);
88 + const analysis = text(value?.analysis);
89 + if (!analysis || !Array.isArray(value?.findings)) return undefined;
90 + const recommendations = Array.isArray(value.recommendations)
91 + ? value.recommendations.filter(
92 + (recommendation): recommendation is string =>
93 + typeof recommendation === "string" && recommendation.length > 0,
94 + )
95 + : [];
96 + return { analysis, findings: value.findings, recommendations };
99 +function researchReport(result: WorkflowResult): ResearchReport | undefined {
100 + if (result.workflow !== "research") return undefined;
101 + const items = resultItems(result.result);
102 + if (items.length !== 1) return undefined;
103 + const value = record(items[0]);
104 + const answerStatus = text(value?.answer_status);
105 + const executiveSummary = text(value?.executive_summary);
106 + const analysis = text(value?.analysis);
107 + if (
108 + !answerStatus ||
109 + !executiveSummary ||
110 + !analysis ||
111 + !Array.isArray(value?.findings) ||
112 + !Array.isArray(value.sources)
114 + return undefined;
115 + return {
116 + answerStatus,
117 + executiveSummary,
118 + analysis,
119 + findings: value.findings,
120 + contradictions: stringArray(value.contradictions),
121 + gaps: stringArray(value.gaps),
122 + recommendations: stringArray(value.recommendations),
123 + sources: value.sources,
124 + };
127 +function researchFindingLine(value: unknown): string {
128 + const finding = record(value);
129 + if (!finding) return itemLine(value);
130 + const claim = text(finding.claim) ?? JSON.stringify(value);
131 + const evidence = text(finding.evidence);
132 + const confidence = text(finding.confidence);
133 + const citations = stringArray(finding.citations);
134 + return [
135 + `- **${claim}**${confidence ? ` [${confidence}]` : ""}`,
136 + ...(evidence ? [` ${evidence}`] : []),
137 + ...(citations.length > 0 ? [` Sources: ${citations.join(", ")}`] : []),
138 + ].join("\n");
141 +function researchSourceLine(value: unknown): string {
142 + const source = record(value);
143 + if (!source) return itemLine(value);
144 + const title = text(source.title) ?? "Untitled source";
145 + const url = text(source.url);
146 + const sourceType = text(source.source_type);
147 + const publishedAt = text(source.published_at);
148 + const metadata = [sourceType, publishedAt].filter(Boolean).join(" · ");
149 + return `- **${title}**${metadata ? ` (${metadata})` : ""}${url ? ` — ${url}` : ""}`;
58 152 function refutedSlots(result: WorkflowResult): PhaseSlotResult[] {
59 153 return Object.values(result.phaseResults)
60 154 .flat()
157 251 result: WorkflowResult,
158 252 expanded = false,
159 253 ): string {
160 - const finalItems = resultItems(result.result);
254 + const report = reviewReport(result);
255 + const research = researchReport(result);
256 + const finalItems =
257 + report?.findings ?? research?.findings ?? resultItems(result.result);
161 258 const lines = [
162 259 `# ultra ${result.workflow}${result.steered ? " (steered)" : ""}`,
176 273 (total, phase) => total + phase.dropped,
179 - lines.push(
180 - "## Summary",
181 - summaryLine(result, finalItems.length, refuted.length, droppedCount),
182 - "",
183 - );
276 + if (report) lines.push("## Analysis", report.analysis, "");
277 + else if (research)
278 + lines.push(
279 + "## Answer",
280 + research.executiveSummary,
281 + `Status: ${research.answerStatus}`,
282 + "",
283 + "## Analysis",
284 + research.analysis,
285 + "",
286 + );
287 + else
288 + lines.push(
289 + "## Summary",
290 + summaryLine(result, finalItems.length, refuted.length, droppedCount),
291 + "",
292 + );
185 294 if (finalItems.length > 0) {
186 295 lines.push(
187 296 result.workflow === "review"
188 297 ? "## Verified findings"
189 - : "## Final results",
190 - ...finalItems.map(itemLine),
298 + : research
299 + ? "## Findings"
300 + : "## Final results",
301 + ...(research
302 + ? finalItems.map(researchFindingLine)
303 + : finalItems.map(itemLine)),
316 + if (research?.contradictions.length)
317 + lines.push(
318 + "## Contradictions",
319 + ...research.contradictions.map((item) => `- ${item}`),
320 + "",
321 + );
322 + if (research?.gaps.length)
323 + lines.push(
324 + "## Research gaps",
325 + ...research.gaps.map((item) => `- ${item}`),
326 + "",
327 + );
203 329 const drops = dropLines(result, expanded);
204 330 if (drops.length > 0) lines.push("## Drops", ...drops, "");
205 331 lines.push(
206 332 "## Recommendations",
207 - ...recommendationLines(result, finalItems.length, droppedCount),
333 + ...(report?.recommendations.length
334 + ? report.recommendations.map((recommendation) => `- ${recommendation}`)
335 + : research?.recommendations.length
336 + ? research.recommendations.map(
337 + (recommendation) => `- ${recommendation}`,
339 + : recommendationLines(result, finalItems.length, droppedCount)),
342 + if (research?.sources.length)
343 + lines.push("## Sources", ...research.sources.map(researchSourceLine), "");
211 345 if (expanded)
212 346 lines.push(
221 355 export function formatWorkflowResultMessage(result: WorkflowResult): string {
222 - if (result.workflow === "review") return formatUltraResultMarkdown(result);
356 + if (result.workflow === "review" || result.workflow === "research")
357 + return formatUltraResultMarkdown(result);
223 358 return [
224 359 `ultra workflow "${result.workflow}" result${result.steered ? " (steered)" : ""}:`,
225 360 "```json",
228 363 ].join("\n");
231 -export function formatBackgroundAcceptedMessage(
232 - runId: string,
233 - workflowName: string,
234 -): string {
235 - return `ultra background run "${workflowName}" started: ${runId}`;
238 -export function formatBackgroundResultMessage(
239 - runId: string,
240 - result: WorkflowResult,
241 -): string {
242 - return [
243 - `ultra background run "${result.workflow}" complete: ${runId}`,
244 - formatWorkflowResultMessage(result),
245 - ].join("\n\n");
248 366 // ---------------------------------------------------------------------------
249 367 // Tool-surface progress sink
250 368 // ---------------------------------------------------------------------------
338 456 return items.length > 0 ? items : null;
459 +const ULTRA_SUBCOMMANDS: AutocompleteItem[] = [
461 + value: "exec",
462 + label: "exec",
463 + description: "execute instructions through an authored workflow",
464 + },
466 + value: "run",
467 + label: "run",
468 + description: "run a saved workflow",
469 + },
472 +export function completeUltraCommand(
473 + discovered: readonly DiscoveredWorkflow[],
474 + prefix: string,
475 +): AutocompleteItem[] | null {
476 + const input = prefix.trimStart();
477 + if (!/\s/.test(input)) {
478 + const p = input.toLowerCase();
479 + const items = ULTRA_SUBCOMMANDS.filter((item) => item.value.startsWith(p));
480 + return items.length > 0 ? items : null;
482 + const match = input.match(/^run\s+(.*)$/i);
483 + if (!match) return null;
484 + const workflows = completeWorkflowNames(discovered, match[1]);
485 + return (
486 + workflows?.map((item) => ({
487 + ...item,
488 + value: `run ${item.value}`,
489 + })) ?? null
490 + );
493 +export type UltraCommand =
494 + | { subcommand: "activate" }
495 + | { subcommand: "exec"; instructions: string }
496 + | { subcommand: "run"; invocation: string }
497 + | { subcommand: "invalid" };
499 +export function parseUltraCommand(raw: string): UltraCommand {
500 + const input = raw.trim();
501 + if (!input) return { subcommand: "activate" };
502 + const space = input.search(/\s/);
503 + const subcommand = space === -1 ? input : input.slice(0, space);
504 + const rest = space === -1 ? "" : input.slice(space + 1).trim();
505 + if (subcommand === "exec" && rest)
506 + return { subcommand: "exec", instructions: rest };
507 + if (subcommand === "run" && rest)
508 + return { subcommand: "run", invocation: rest };
509 + return { subcommand: "invalid" };
341 512 // ---------------------------------------------------------------------------
342 513 // Spec resolution
343 514 // ---------------------------------------------------------------------------
345 -/** The `workflow` tool / `/ultra` command input. */
516 +/** The `workflow` tool / `/ultra run` command input. */
346 517 export interface WorkflowInput {
347 518 name?: string;
348 519 spec?: unknown;
353 - * Parse a raw `/ultra <name> [args]` command line: the first whitespace token is
524 + * Parse a raw `/ultra run <name> [args]` invocation: the first whitespace token is
354 525 * the workflow name; the remainder is either a trailing JSON object (used as the
355 526 * run args directly) or bare text returned verbatim as `rest` for positional
356 527 * mapping against the spec's declared `args.params` (see `mapPositionalArgs`).
388 559 * positional `params`: whitespace-split tokens fill `params` in order, and the
389 560 * last declared param soaks any remaining tokens so a trailing free-text arg
390 561 * (e.g. a feature description) survives intact. Returns `undefined` when nothing
391 - * maps (empty remainder or no declared params). Only the `/ultra` command uses
562 + * maps (empty remainder or no declared params). Only `/ultra run` uses
392 563 * this — the `workflow` tool passes a structured `args` object directly.
394 565 export function mapPositionalArgs(
439 610 // Validate the discovered spec at resolve time (not at discovery, so a
440 - // malformed file still lists in the picker) — `/ultra <name>` then fails
611 + // malformed file still lists in the picker) — `/ultra run <name>` then fails
441 612 // fast with parseWorkflow's TypeBox path error instead of crashing the
442 613 // engine mid-run on a raw, unvalidated object.
443 614 const spec = parseWorkflow(found.spec);
444 615 let args = input.args;
445 - // Positional fallback: a bare `/ultra <name> <tok...>` line (no JSON args
616 + // Positional fallback: a bare `/ultra run <name> <tok...>` line (no JSON args
446 617 // object) maps to the spec's declared `args.params` in order.
447 618 if (args === undefined && rest && spec.args?.params?.length) {
448 619 args = mapPositionalArgs(rest, spec.args.params);

extensions/ultra/workflows/research.json

Added
new file mode 100644
1 +{
2 + "name": "research",
3 + "description": "Research a generic question through parallel evidence gathering, gap analysis, adversarial verification, and cited synthesis.",
4 + "args": { "hint": "<question>", "params": ["question"] },
5 + "schemas": {
6 + "ResearchPlan": {
7 + "type": "object",
8 + "properties": {
9 + "question": { "type": "string" },
10 + "answer_shape": { "type": "string" },
11 + "angles": {
12 + "type": "array",
13 + "minItems": 3,
14 + "maxItems": 4,
15 + "items": {
16 + "type": "object",
17 + "properties": {
18 + "id": { "type": "string" },
19 + "question": { "type": "string" },
20 + "source_strategy": { "type": "string" },
21 + "opposition_strategy": { "type": "string" }
22 + },
23 + "required": [
24 + "id",
25 + "question",
26 + "source_strategy",
27 + "opposition_strategy"
28 + ]
29 + }
30 + }
31 + },
32 + "required": ["question", "answer_shape", "angles"]
33 + },
34 + "Evidence": {
35 + "type": "object",
36 + "properties": {
37 + "angle_id": { "type": "string" },
38 + "angle": { "type": "string" },
39 + "findings": {
40 + "type": "array",
41 + "items": {
42 + "type": "object",
43 + "properties": {
44 + "claim": { "type": "string" },
45 + "evidence": { "type": "string" },
46 + "source_url": { "type": "string" },
47 + "source_title": { "type": "string" },
48 + "source_type": { "type": "string" },
49 + "published_at": { "type": "string" },
50 + "confidence": { "type": "string" }
51 + },
52 + "required": [
53 + "claim",
54 + "evidence",
55 + "source_url",
56 + "source_title",
57 + "source_type",
58 + "confidence"
59 + ]
60 + }
61 + },
62 + "contradictions": { "type": "array", "items": { "type": "string" } },
63 + "gaps": { "type": "array", "items": { "type": "string" } },
64 + "leads": { "type": "array", "items": { "type": "string" } }
65 + },
66 + "required": [
67 + "angle_id",
68 + "angle",
69 + "findings",
70 + "contradictions",
71 + "gaps",
72 + "leads"
73 + ]
74 + },
75 + "Audit": {
76 + "type": "object",
77 + "properties": {
78 + "assessment": { "type": "string" },
79 + "verification_tasks": {
80 + "type": "array",
81 + "maxItems": 3,
82 + "items": {
83 + "type": "object",
84 + "properties": {
85 + "id": { "type": "string" },
86 + "claim": { "type": "string" },
87 + "why": { "type": "string" },
88 + "preferred_sources": { "type": "string" },
89 + "opposition_query": { "type": "string" }
90 + },
91 + "required": [
92 + "id",
93 + "claim",
94 + "why",
95 + "preferred_sources",
96 + "opposition_query"
97 + ]
98 + }
99 + },
100 + "unresolved_gaps": { "type": "array", "items": { "type": "string" } }
101 + },
102 + "required": ["assessment", "verification_tasks", "unresolved_gaps"]
103 + },
104 + "Verification": {
105 + "type": "object",
106 + "properties": {
107 + "id": { "type": "string" },
108 + "claim": { "type": "string" },
109 + "verdict": { "type": "string" },
110 + "explanation": { "type": "string" },
111 + "sources": {
112 + "type": "array",
113 + "items": {
114 + "type": "object",
115 + "properties": {
116 + "title": { "type": "string" },
117 + "url": { "type": "string" },
118 + "source_type": { "type": "string" }
119 + },
120 + "required": ["title", "url", "source_type"]
123 + },
124 + "required": ["id", "claim", "verdict", "explanation", "sources"]
125 + },
126 + "ResearchReport": {
127 + "type": "object",
128 + "properties": {
129 + "answer_status": { "type": "string" },
130 + "executive_summary": { "type": "string" },
131 + "analysis": { "type": "string" },
132 + "findings": {
133 + "type": "array",
134 + "items": {
135 + "type": "object",
136 + "properties": {
137 + "claim": { "type": "string" },
138 + "evidence": { "type": "string" },
139 + "confidence": { "type": "string" },
140 + "citations": { "type": "array", "items": { "type": "string" } }
141 + },
142 + "required": ["claim", "evidence", "confidence", "citations"]
144 + },
145 + "contradictions": { "type": "array", "items": { "type": "string" } },
146 + "gaps": { "type": "array", "items": { "type": "string" } },
147 + "recommendations": { "type": "array", "items": { "type": "string" } },
148 + "sources": {
149 + "type": "array",
150 + "items": {
151 + "type": "object",
152 + "properties": {
153 + "title": { "type": "string" },
154 + "url": { "type": "string" },
155 + "source_type": { "type": "string" },
156 + "published_at": { "type": "string" }
157 + },
158 + "required": ["title", "url", "source_type"]
161 + },
162 + "required": [
163 + "answer_status",
164 + "executive_summary",
165 + "analysis",
166 + "findings",
167 + "contradictions",
168 + "gaps",
169 + "recommendations",
170 + "sources"
173 + },
174 + "phases": [
176 + "id": "plan",
177 + "kind": "single",
178 + "step": {
179 + "summary": "Plan research for: {args.question}",
180 + "prompt": "Reframe this research request into the actual question or decision it must answer: {args.question}. Define the most useful answer shape, then produce 3 to 4 orthogonal research angles. Each angle must have a stable short id, a focused question, a source strategy favoring primary and authoritative evidence, and an opposition strategy for finding disconfirming evidence. Avoid overlapping angles. Return JSON matching ResearchPlan.",
181 + "model": "big",
182 + "thinkingLevel": "high",
183 + "schema": "ResearchPlan"
185 + },
187 + "id": "gather",
188 + "kind": "fanout",
189 + "over": "{plan.results[].angles[]}",
190 + "step": {
191 + "summary": "Gather evidence for research angle {item}",
192 + "prompt": "Research this angle for the broader question: {args.question}. Angle: {item}. Do not modify local files or remote systems. Use at least two meaningfully different web searches. Prefer primary sources, official documentation, papers, direct data, and reputable independent analysis. Fetch and read the two strongest sources rather than relying on snippets. Keep each fetch targeted with linesMatching or maxChars no greater than 8000; do not dump whole large pages into context. Seek contrary evidence using the angle's opposition strategy. Record one atomic finding per source-backed claim with exact URL, title, source type, publication date when available, and confidence based on source quality and corroboration. Explicitly return contradictions, gaps, and promising leads. Never fabricate a URL or claim. Echo the angle id and question. Return JSON matching Evidence.",
193 + "tools": ["web_search", "web_fetch", "web_map"],
194 + "model": "small",
195 + "thinkingLevel": "medium",
196 + "schema": "Evidence"
198 + },
200 + "id": "audit",
201 + "kind": "single",
202 + "step": {
203 + "summary": "Audit evidence and identify verification targets",
204 + "prompt": "Audit the gathered evidence for this question: {args.question}. Research plan: {plan.results}. Evidence by angle: {gather.results}. Compare claims across angles and sources. Identify material contradictions, important single-source claims, source monocultures, stale evidence, circular citations, and decision-relevant gaps. Create at most 3 concrete verification tasks whose resolution could materially change the answer; prioritize contradictions and high-impact weak claims. Each task needs a stable id, exact claim, why it matters, preferred source types, and an opposition query. Put gaps that cannot be verified into unresolved_gaps. Return JSON matching Audit.",
205 + "model": "big",
206 + "thinkingLevel": "high",
207 + "schema": "Audit"
209 + },
211 + "id": "verify",
212 + "kind": "fanout",
213 + "when": "{audit.results[].verification_tasks[]}",
214 + "over": "{audit.results[].verification_tasks[]}",
215 + "step": {
216 + "summary": "Verify research claim {item}",
217 + "prompt": "Adversarially verify this research task for the broader question: {args.question}. Task: {item}. Do not modify local files or remote systems. Search for primary, authoritative, independent, and disconfirming evidence. Fetch at most 3 substantive sources and inspect what they actually support. Keep each fetch targeted with linesMatching or maxChars no greater than 8000; do not dump whole large pages into context. Use one verdict string: supported, refuted, qualified, or unresolved. Echo the task id and claim exactly. Explain the verdict and return every source used with title, URL, and source type. Never fabricate a URL. Return JSON matching Verification.",
218 + "tools": ["web_search", "web_fetch", "web_map"],
219 + "model": "big",
220 + "thinkingLevel": "high",
221 + "schema": "Verification"
223 + },
225 + "id": "report",
226 + "kind": "single",
227 + "step": {
228 + "summary": "Synthesize cited research answer",
229 + "prompt": "Write the final research answer for: {args.question}. Plan: {plan.results}. Gathered evidence: {gather.results}. Evidence audit: {audit.results}. Verification verdicts: {verify.results}. Lead with a direct executive answer and state answer_status as fully answered, partially answered, or unresolved. Synthesize by theme and question rather than source order. Include only claims traceable to supplied URLs, calibrate confidence from source quality, independence, agreement, and verification, and preserve material qualifications. Explain contradictions and whether they were resolved. List remaining gaps and concrete next research or decision steps. Deduplicate the source registry. Never invent or alter URLs and never dump source JSON into prose. Return JSON matching ResearchReport.",
230 + "model": "big",
231 + "thinkingLevel": "high",
232 + "schema": "ResearchReport"
235 + ],
236 + "return": "{report.results}"

extensions/ultra/workflows/review.json

Modified
2 2 "name": "review",
3 - "description": "Review a PR, branch, ref, recent changes, folder, or file; then adversarially verify each finding.",
3 + "description": "Review a PR, branch, ref, recent changes, folder, or file; verify each finding; then synthesize analysis and next steps.",
4 4 "args": { "hint": "[target]", "params": ["target"] },
5 5 "schemas": {
6 6 "Findings": {
35 35 "required": ["title", "real", "file"]
36 + },
37 + "ReviewReport": {
38 + "type": "object",
39 + "properties": {
40 + "analysis": { "type": "string" },
41 + "findings": {
42 + "type": "array",
43 + "items": {
44 + "type": "object",
45 + "properties": {
46 + "title": { "type": "string" },
47 + "file": { "type": "string" },
48 + "why": { "type": "string" },
49 + "severity": { "type": "string" }
50 + },
51 + "required": ["title", "file", "why", "severity"]
52 + }
53 + },
54 + "recommendations": {
55 + "type": "array",
56 + "items": { "type": "string" }
57 + }
58 + },
59 + "required": ["analysis", "findings", "recommendations"]
38 62 "phases": [
60 84 "thinkingLevel": "high",
61 85 "schema": "Verdict"
87 + },
88 + {
89 + "id": "report",
90 + "kind": "single",
91 + "step": {
92 + "summary": "Synthesize review analysis and next steps",
93 + "prompt": "Write the final human-readable review report for target: {args.target}. Candidate scans: {review.results}. Verification verdicts: {verify.results}. Include only verdicts with real=true in findings. Explain the overall code state in analysis, including material strengths, gaps, and review confidence. Give concrete ordered recommendations that move the work forward; when no finding is verified, recommend the most relevant validation or readiness step instead of generic filler. Keep prose concise and never dump source JSON into strings. Return JSON matching ReviewReport.",
94 + "model": "big",
95 + "thinkingLevel": "medium",
96 + "schema": "ReviewReport"
97 + }
65 - "return": "{verify.results | where real}"
100 + "return": "{report.results}"