repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by admin

feat(ultra): separate workflow result audiences

b84cacf52cdfa8879a0f311aeac3b93d8cbdfd81

Oliver Jan Krylow <oliver@bugabinga.net> ·

SSH signature · not checked

Browse filesCompare with HEAD

Changed files (25)

CHANGELOG.md

Modified
2 2 # Changelog
4 4 ## Unreleased
6 +- Refine Ultra model-tier selection around residual judgement: default to `small`, use `tiny` only for deterministic work, decompose when context or risk benefits exceed handoff cost, and block rather than compensate for missing facts or authority
6 7 - Add Witness definition discovery, validation, trusted project overrides, `/witness list`, `/witness show`, `/witness validate`, and isolated manual text observers through `/witness run <id>`; cancel stale runs and preserve child usage
7 8 - Load no extensions into Angel children by default, add `angel.subagentExtensions` for `"all"`, `"none"`, or a static sibling-extension whitelist, and keep unavailable parent-local or on-demand tools from blocking consultation
8 9 - Keep footer segment order stable when multiple segments share the same priority
65 66 - Fix Strata cancellation and session shutdown so pending requests and subprocesses settle before cleanup finishes
66 67 - Keep Ultra's intermediate results out of orchestrator context, return final output once, and replace output above 50KB or 2000 lines with a full-output file reference while preserving complete results in UI details
67 68 - Let Ultra configure named model tiers with default thinking levels and orchestrator selection guidance, merge global and project overrides, use the session model by default across providers, and start fresh checkpoints when effective model or thinking routes change
68 -- Make Ultra choose the least-compute model tier likely to meet each step's acceptance criteria and escalate only for a concrete reasoning bottleneck
69 69 - Remove Strata Ask's fixed tool-call, round, batch, and cumulative-output caps while retaining per-tool resource limits and cancellation
70 70 - Add optional Strata GitHub PR reviews with confirmed checkout switching, attributed read-only discussions, and PR-aware feedback; keep saved plans and drafts separate across PRs and forge hosts
71 71 - Add Strata Vim-style line, hunk, layer, and pane navigation, literal diff search, keyboard review actions, whole-layer review toggles, and contextual shortcut guides

extensions/ultra/README.md

Modified
8 8 ## Two surfaces
10 10 - **`/ultra`**: enables the hidden `run_workflow` tool for later agent turns, confirms activation, and does not start a turn.
11 -- **`/ultra exec <instructions>`**: enables the tool, sends the request with a short exec-specific instruction as a visible user message, and starts an agent turn.
11 +- **`/ultra exec <instructions>`**: enables the tool, sends the request as a visible user message, and starts an agent turn.
12 12 - **`/ultra run <name> [args]`**: runs a saved workflow behind a phase-first live overlay with keyboard phase toggling, agent focus, streaming transcripts, steering, targeted abort, and confirmed whole-run abort.
13 13 - **`run_workflow` tool**: model-callable after `/ultra` enables it. Pass a saved
14 14 workflow `name` (+ optional `args`), or an inline `spec` object for a one-off
18 18 ## Workflows
20 -Use ultra only when independent parallel work, staged agent handoffs, or isolated specialist contexts materially improve the result.
21 -Otherwise perform the task directly instead of wrapping it in ultra.
23 20 A workflow is a single JSON object — phases (`single` / `fanout`), per-step
24 21 prompts/tools/schemas, and a closed interpolation grammar that pipes one phase's
25 22 results into the next. The authoritative format reference is
27 24 - [`prompts/references/spec-format.md`](prompts/references/spec-format.md)
29 -— the single source of truth, kept in lockstep with the runtime validator by a
30 -test. Don't reconstruct the format from memory; read it.
26 +— the canonical authoring reference, kept in lockstep with the runtime validator by a
27 +test. The runtime schema is also exposed on the active tool. Don't reconstruct the format from memory; read it.
32 29 ### Authoring
34 31 The active `run_workflow` tool advertises the [Ultra authoring skill](prompts/ultra-authoring/SKILL.md) for ordinary tool use and `/ultra exec`.
35 -Read it once before authoring/editing workflow specs or first calling the tool; it covers persistence, bounded handoffs, fanout ownership, schemas, interpolation, dynamic extensions, and tier selection.
32 +Read it once before authoring/editing workflow specs or first calling the tool; it explains command routing, workflow mechanics, child context, dynamic extensions, results, and model fields.
36 33 The skill lives outside automatic skill discovery: no global catalog entry or `/skill` command, and no advertised path while the workflow tool is inactive.
37 34 Manual `/ultra` activation exposes the path immediately, without a reload; deactivation removes the guidance, not previously read conversation history.
38 -`/ultra exec` sends only its short instruction and the user's request; it does not inject guides or examples.
35 +`/ultra exec` sends only the user's request; the active tool supplies the skill path through `promptGuidelines`.
39 36 The [canonical spec reference](prompts/references/spec-format.md) provides detailed field semantics and examples.
41 -The provider-facing schema accepts an inline `spec` object without embedding workflow fields; the canonical reference supplies their full format on demand.
42 -Both command and tool invocations still pass through the unchanged strict runtime validator, including conditional fanout and write-isolation requirements.
43 -The effective model-tier data remains dynamically injected while the workflow tool is active; selection rules live in the authoring skill.
38 +The provider-facing schema exposes the structural workflow fields and descriptions for inline `spec` objects.
39 +Both command and tool invocations still pass through the strict runtime validator, including conditional fanout and write-isolation requirements that JSON Schema cannot express.
40 +The effective model-tier data remains dynamically injected while the workflow tool is active.
44 41 Each compact entry preserves the configured model, default thinking level, and selection guidance.
46 43 `mise run //extensions/ultra:authoring-check` runs a read-only planning request using the configured model.
94 91 For both tool and command results, the full aggregate is always stored once and every bounded model-facing envelope includes `fullOutputPath`.
95 92 The envelope contains the selected final value when it fits, phase counts, bounded failure diagnostics, and usage, not duplicated `phaseResults`.
93 +When `report` is configured, it is persisted in the full aggregate and rendered only on human-facing surfaces; it is omitted from the bounded model-facing envelope.
96 94 `executionStatus` is `finished`, `incomplete` for dropped steps, or `aborted`; `finished` describes execution only, not artifact quality or semantic success.
97 95 Oversized values use a retrieval notice within the 50KB or 2000 lines limit; read the saved aggregate selectively.
98 96 Complete contents, including intermediate results and failure inputs/transcript tails, remain in UI details.
112 110 The overlay also shows each exact resolved sub-agent prompt as the first user message in its transcript.
113 111 Sub-agent tool calls keep one native Pi tool row from invocation through result and follow Pi's global expand state.
114 112 Collapsed workflow tool output shows counts, reported cost, and output tokens with Pi's standard expand hint.
113 +Interactive completion also notifies the user with the status, phase count, and usage; headless runs stay silent.
115 114 Expanded tools and overlays add the uncached-input, cache-read, and cache-write breakdown.
116 -Expanded tool output summarizes every phase, automatically opens the active and failed phases, shows their agents, and ends with the human-readable workflow outcome.
115 +Expanded tool output summarizes every phase, automatically opens the active and failed phases, shows their agents, and ends with the configured human-readable report when present.
117 116 The command overlay uses the same board, automatically collapses completed phases, and adds keyboard interaction.
118 117 Use `↑`/`↓` to focus visible phase or agent rows, `Enter` to toggle a phase, and `Tab` to cycle visible agents.
119 118 Use `Alt+↑`/`Alt+↓` to scroll the focused transcript, `s` to steer, and `x` to abort the focused agent.
122 121 Rows transition through queued, running, done, dropped, or skipped without disappearing from their phase summary.
123 122 Configured tiers are shown as `tier → provider/model`.
125 -Workflow results lead with reported cost and output tokens from completed sub-agent turns.
126 -Input/cache details appear on expansion; summed tokens are not presented as context size or the primary usage metric.
124 +Completion notifications report cost and output tokens from completed sub-agent turns.
125 +Configured human reports do not include usage; notifications and persisted JSON retain usage and input/cache details.
127 126 The final JSON contains `tokenUsage` with `input`, `output`, `total`, `cacheRead`, `cacheWrite`, and `cost`.
128 127 Totals count every assistant response from launched steps exactly once, including result-submission reminders and failed or aborted attempts; reused checkpoints add no new usage.
169 168 A new tier must provide all three fields; global and project settings may partially override an existing tier field by field.
170 169 Validate a settings file with `mise run //extensions/ultra:check-settings -- /path/to/settings.json`.
172 -Built-in model-specific tier responsibilities and thinking defaults live only in [`settings.ts`](settings.ts), not in a duplicated fixed tier-role table.
173 -The [delegation contract](prompts/ultra-authoring/SKILL.md#delegation-contract) governs instruction preservation, context sufficiency, and acceptable artifacts.
171 +Built-in model tier defaults live in [`settings.ts`](settings.ts).
174 172 When `run_workflow` is active, Ultra injects one effective rubric generated from the merged tiers into the orchestrator system prompt.
175 -That applies to ordinary tool use and `/ultra exec`.
176 -The rubric, not a hardcoded role list in an authored workflow, decides tier selection.
177 -The orchestrator chooses the least-compute tier likely to satisfy each step's acceptance criteria and escalates only for a concrete remaining reasoning bottleneck.
178 -Broad scope, long prompts, routine review or debugging, and general importance do not justify more compute; missing context is supplied or reported blocked instead.
179 173 Tier instructions guide the orchestrator only and are never passed to child agents as model-specific instructions.
181 175 A step's `model` may name any configured tier or give a concrete `provider/model` reference.
182 176 An unknown bare tier name is an error; a concrete provider/model reference remains valid.
183 177 Omitting `model`, or resolving a tier whose `model` is `null`, uses the session model.
184 -An explicit step `thinkingLevel` overrides the tier default, but authors normally omit it so settings control the budget.
178 +An explicit step `thinkingLevel` overrides the tier default.
185 179 Saved workflows retain their explicit tier selections when settings change.
186 180 Changing effective model or thinking routes starts a fresh checkpoint journal instead of resuming children with stale routes.
187 181 Changes only to the tier's orchestrator-facing selection instructions do not invalidate execution checkpoints.

extensions/ultra/U008.md

Deleted
deleted file mode 100644
1 -# U008 Generated subagent extensions
3 -## Status
5 -Implemented.
7 -This document is binding for U008.
8 -If implementation pressure contradicts it, change this specification explicitly before changing behavior.
10 -## Outcome
12 -Ultra may delegate creation, reuse, improvement, validation, and repair of purpose-built Pi extensions to builder subagents.
13 -Validated dynamic extensions load into later child sessions through each child's independent `ResourceLoader` and behave like normal extensions there.
15 -The feature optimizes the user's task, not extension authoring itself.
16 -Ultra should encourage reusable dynamic extensions when they can encode specialized capability, deterministic operations, or repeated reasoning, because the shared catalog amortizes authoring cost across workflows.
18 -## Trust boundary
20 -Dynamic extensions have the same authority as the orchestrator that authored them.
21 -They are not sandboxed and may use the complete Pi extension API and host permissions.
22 -Validation process isolation contains crashes, hangs, leaked handles, and mutated globals; it is not a security boundary.
24 -Builders receive every available child tool, including loaded extension tools, except `run_workflow`.
25 -Ultra's recursion boundary remains unchanged.
27 -## Storage
29 -### Durable catalog
31 -The catalog lives beneath Pi's agent directory:
33 -```text
34 -<agentDir>/ultra/dynamic-extensions/catalog/
35 - <name>/
36 - current.json
37 - <sha256>/
38 - manifest.json
39 - package.json
40 - README.md
41 - index.ts
42 - index.test.ts
43 - benchmark.ts
44 - debug.ts
45 - .ultra-validation.json
46 - ...
47 -```
49 -`<name>` is stable extension identity.
50 -`current.json` selects the one canonical validated source directory for that name.
51 -Search indexes only the selected source, never non-current hash directories.
53 -Ultra does not initialize, add, commit, or otherwise manage any version-control repository around `agentDir`.
54 -The catalog may itself be tracked by the user.
56 -### Platform cache
58 -Transient state lives beneath the platform cache directory:
60 -- Windows: `%LOCALAPPDATA%/pi/ultra/dynamic-extensions/`
61 -- macOS: `~/Library/Caches/pi/ultra/dynamic-extensions/`
62 -- Other platforms: `$XDG_CACHE_HOME/pi/ultra/dynamic-extensions/`, falling back to `~/.cache/pi/ultra/dynamic-extensions/`
64 -The cache contains:
66 -```text
67 -runs/<parent-session-id>/<workflow-run-id>/<agent-id>/
68 -locks/<catalog-path-hash>/<extension-name>.lock
69 -logs/<child-session-id>/<extension-name>.jsonl
70 -```
72 -Completed workflows delete drafts.
73 -Interrupted workflows retain drafts for resume.
74 -Abandoned drafts are lazily pruned after seven days.
75 -Completed and interrupted debug logs are retained for seven days.
76 -No cleanup runs during ordinary Ultra startup.
78 -## Canonical publication
80 -The source hash is SHA-256 over sorted relative paths and exact bytes of every artifact file, including documentation, tests, package metadata, benchmarks, and `debug.ts`.
81 -Only `.ultra-validation.json` is excluded.
82 -Symlinks are rejected.
83 -There is no file-count or artifact-size limit.
85 -Cross-platform APIs cannot atomically replace an existing non-empty directory.
86 -Publication therefore writes a new immutable hash directory and atomically replaces the small `current.json` file.
87 -Publication points `current.json` at the new source only after the source and attestation are complete.
88 -Non-current source directories are excluded from discovery and may be removed after publication; crash leftovers are pruned lazily.
90 -A per-name cross-process lock protects publication.
91 -Locks live in platform cache so a version-controlled `agentDir` receives no transient files.
92 -Publication compares the draft's recorded base hash with current canonical state.
93 -A stale base is rejected, never overwritten or textually auto-merged.
94 -The builder receives base/current hashes and paths, checks out current canonical state, reapplies its intent, and validates again.
95 -One stale-base retry is recommended, but not enforced by the runtime.
97 -Validation publishes immediately, before consumer use.
98 -A validated extension that is never ultimately loaded may therefore remain canonical; this is an accepted simplification.
100 -Manual edits to canonical source invalidate its attested hash.
101 -Search marks it unvalidated, and loading reports the exact path plus an instruction to call `validate_dynamic_extension`.
102 -Validation accepts both tool-created drafts and manually edited canonical directories.
103 -Canonical validation locks and republishes the edited source under its new hash.
105 -Before child creation, Ultra resolves each stable name to current validated hash while holding the short name lock and copies that exact source into a run-local hash snapshot.
106 -The child loads the snapshot, not mutable catalog state.
107 -This prevents publication races between resolution and `ResourceLoader` execution.
109 -## Builder tools
111 -Ultra supplies four strict-schema tools to builder children:
113 -- `search_dynamic_extensions`
114 -- `create_dynamic_extension`
115 -- `checkout_dynamic_extension`
116 -- `validate_dynamic_extension`
118 -A workflow step selecting any builder tool must select all four.
119 -That complete set marks the child as a builder and injects the versioned builder contract.
120 -No search-only child role exists.
121 -Consumers receive no dynamic-extension instructions; loaded dynamic extensions look normal to them.
123 -### Search
125 -`search_dynamic_extensions` requires a query and returns at most five deterministic matches.
126 -It tokenizes without a model: lowercase normalization, punctuation and identifier splitting, and simple plural normalization.
127 -Ranking weights exact phrase, name/topic, tags, scope, then the README Purpose, Scope, and Usage sections.
128 -Code blocks and remaining README prose are not indexed.
129 -Ties sort deterministically by name.
131 -Results contain stable name, current hash, validation status, short description, topic, tags, tested-platform advisory metadata, and exact current source path.
132 -Platform metadata never filters or lowers thematic rank.
133 -The builder reads only relevant candidate READMEs before deciding reuse, checkout, or creation.
135 -`create_dynamic_extension` requires a prior search call in the same builder session.
136 -The tool records returned candidates and requires explicit rejection rationale, including the no-match case, before creating a new topic.
138 -### Create
140 -`create_dynamic_extension` requires:
142 -- name
143 -- description
144 -- topic
145 -- scope
146 -- non-goals
147 -- tags
148 -- considered candidates and rejection rationale
150 -It scaffolds an editable run-cache draft and returns its absolute path.
151 -The initial README is populated from these fields rather than empty headings.
152 -Validation rejects unresolved placeholders.
154 -### Checkout
156 -`checkout_dynamic_extension` copies current canonical source into an editable run-cache draft and records its base hash.
157 -Normal tools edit the returned absolute path.
158 -Name and topic are immutable identity during checkout.
159 -Scope, non-goals, tags, API, docs, tests, benchmarks, and implementation may evolve within that topic.
160 -A different topic requires a new extension.
162 -### Validate
164 -`validate_dynamic_extension` runs mandatory correctness gates and advisory performance checks.
165 -Correctness failure retains the draft and publishes nothing.
166 -Performance budget overruns produce warnings and do not block publication because scheduler load is not correctness evidence.
167 -Success returns stable name, full revision hash, description, and exact canonical source path.
169 -Diagnostics report all independent failures in one call.
170 -Each diagnostic contains stable gate and code identifiers, path and line where available, expected and actual values, bounded output, elapsed limits, remediation, and retained draft path.
171 -Dependent skipped gates are reported as blocked rather than silently omitted.
173 -Esc abort kills the validator process tree, retains the draft, publishes nothing, and returns a typed aborted result.
175 -## Builder orchestration
177 -The root orchestrator receives concise guidance about when to delegate a builder.
178 -Only builders receive the full authoring contract.
179 -Pi API documentation and example locations already present in Pi's system prompt are not duplicated.
181 -A builder brief is structured as:
183 -- topic, required
184 -- outcome, required
185 -- intended consumers, required
186 -- done conditions, required
187 -- assumptions, optional
188 -- current platform, optional
189 -- relevant paths or evidence, optional
190 -- hard constraints, optional
192 -The orchestrator does not paste transcripts, files, or general Pi documentation into the brief.
193 -Builders inspect relevant sources with their tools.
195 -One builder owns one extension.
196 -Independent topics may use parallel builders when their fanout phase declares write isolation.
197 -Builders should not intentionally modify the same extension in parallel; stale-base validation still prevents clobbering across mistaken plans or separate Pi sessions.
199 -Use `big/high` for new extension architecture and `medium/high` for bounded improvements unless configured models or task economics justify another deliberate choice.
201 -Known canonical names may be loaded directly without a builder.
202 -Builders are required for discovery, creation, improvement, stale validation, and runtime repair.
204 -A builder failure blocks later consumer phases.
205 -Blocking stops later phases while preserving the normal aggregate, typed builder failure, debug-log paths, and usage accounting.
206 -U008 has no normal-tool fallback.
208 -## Workflow schema
210 -### Dynamic extensions
212 -Steps gain `dynamicExtensions`.
213 -It accepts either a literal array of stable catalog names or one existing interpolation selector resolving to such an array.
214 -Consumers interpolate stable names, not hashes.
216 -Immediately before child creation, Ultra resolves each stable name to the exact validated hash and source snapshot.
217 -Resolved name and hash participate in the stable step key.
218 -A repaired canonical extension therefore invalidates only affected checkpoints; unchanged work remains reusable.
220 -One consumer may load multiple dynamic extensions.
221 -Normal configured extensions load first and retain existing behavior.
222 -Dynamic extensions load afterward in declared array order.
223 -`subagentExtensions: "none"` suppresses only normal extensions, not explicit `dynamicExtensions`.
225 -Generated tools must activate themselves during `session_start`, because Ultra cannot name them before loading.
226 -The real-loader validation gate verifies this behavior.
228 -### General mutating fanout
230 -Fanout phases gain optional `writeIsolation`, a non-empty rationale explaining why each item owns disjoint mutation work.
232 -Without `writeIsolation`, existing fanout law remains unchanged: initial tools are read-only plus bash, and the prompt must forbid mutation.
233 -With `writeIsolation`, mutating tools are allowed and the prompt assigns disjoint ownership.
234 -Ultra trusts the orchestrator's declaration.
235 -It does not gate edit/write paths, suppress normal extensions, or attempt to constrain arbitrary custom mutators.
237 -This general fanout relaxation ships with U008.
239 -## Generated extension contract
241 -Each extension covers one topic.
242 -Builders modify an existing extension when the responsibility remains within that topic and create a new extension only for a distinct responsibility.
244 -Names are two-to-four-term kebab-case domain capabilities.
245 -Names contain no platform, version, model, session, `helper`, or `utils` terms.
246 -Descriptions state a concrete capability in one sentence.
248 -The manifest records durable semantics only:
250 -- schema version
251 -- name
252 -- description
253 -- topic
254 -- scope
255 -- non-goals
256 -- tags
257 -- declared capabilities
258 -- required and optional executables
259 -- explicit qualified overrides
260 -- per-platform hot-path benchmark budgets
262 -Provenance, validation results, timings, and source hash belong in `.ultra-validation.json`, not the manifest.
264 -Every declared capability ID appears in the README test matrix and at least one `node:test` case name.
265 -The README contains Purpose, Scope, Non-goals, Alternatives considered, Decision rationale, Usage, Architecture, Test matrix, and Promotion notes.
266 -Alternatives and rationale remain durable documentation and are updated when overlap changes.
268 -Generated extensions have no third-party runtime npm dependencies.
269 -Pi host packages are declared as peer dependencies and added only when imported.
270 -`package.json` remains private until manual promotion.
271 -External executables are allowed when useful; the manifest declares required versus optional tools, and startup checks PATH through native Node APIs.
273 -Generated source and tests use erasable TypeScript supported directly by Pi's Node 22.19+ requirement.
274 -Transform-only enums, namespaces, parameter properties, and equivalent syntax are forbidden.
276 -Entry modules stay thin.
277 -Core logic is exported through pure modules and tested with `node:test`.
278 -Tests use local fakes or loopback services and never external networks.
279 -The real loader separately verifies Pi integration.
281 -Generated extensions must support headless `print` mode.
282 -TUI-only behavior guards TUI APIs, while the capability Ultra needs remains usable headlessly.
284 -The built-in authoring contract requires:
286 -- thin, non-blocking startup
287 -- lazy optional work
288 -- no startup subprocess probes or broad scans
289 -- abort propagation
290 -- bounded tool output
291 -- compact default and expanded detail rendering
292 -- native Node APIs over shell helpers
293 -- no third-party runtime npm dependencies
294 -- typed, actionable errors
295 -- fast tests and benchmarks
296 -- semantic Markdown lines
298 -Loaded extensions must not intentionally modify their own catalog source.
299 -Runtime state uses session entries or explicit project/user data paths; improvements go through checkout and validation.
300 -This is an authoring invariant, not a sandbox restriction.
301 -A later load detects source-hash drift.
303 -Dynamic extensions may explicitly override normal registrations.
304 -The manifest lists exact qualified targets such as `tool:read` or `provider:foo`.
305 -Child composition blocks undeclared collisions with actionable diagnostics.
306 -Normal extensions load before declared dynamic overrides.
308 -## Mandatory scaffold
310 -Every generated extension starts with:
312 -```text
313 -manifest.json
314 -package.json
315 -README.md
316 -index.ts
317 -index.test.ts
318 -benchmark.ts
319 -debug.ts
320 -```
322 -Additional helpers and tests are allowed.
323 -`debug.ts` is copied from Ultra's versioned template.
324 -Builders may call it but validation compares it byte-for-byte and provides a restore instruction when changed.
325 -Validation writes `.ultra-validation.json`; builders do not author that file.
327 -## Validation gates
329 -Validation uses one fresh Node subprocess rooted at the extension draft.
330 -It inherits the host environment and adds only `CI` plus validator temp/log paths.
331 -It has full host permissions.
332 -The parent captures bounded output, applies a hard timeout, and kills the process tree on abort or timeout.
334 -U008 bundles no Biome, Vitest, formatter, linter, or TypeScript compiler.
335 -It installs no dependencies during workflows.
336 -Initial validation therefore includes:
338 -1. manifest schema, naming, one-topic fields, tags, capabilities, executable declarations, override declarations, and benchmark budgets
339 -2. README required sections, populated scaffold content, and capability/test-matrix mapping
340 -3. package constraints and absence of third-party runtime npm dependencies
341 -4. fixed `debug.ts` template
342 -5. symlink rejection and deterministic whole-artifact hashing
343 -6. all `node:test` files with capability mapping
344 -7. required `benchmark.ts` contract and advisory current-platform hot-path budget comparison
345 -8. real Pi SDK loading through scoped `DefaultResourceLoader`, in-memory `AgentSession`, `bindExtensions({ mode: "print" })`, `session_start`, registration inspection, and `session_shutdown`
346 -9. active generated-tool verification
347 -10. advisory marginal extension load timing
348 -11. attestation generation and canonical publication
350 -Real loading uses Pi SDK directly and makes no model call.
351 -`pi-test-harness` is not required.
353 -Five seconds is a warning target, not a quality gate.
354 -Fifteen seconds is the hard validator watchdog.
355 -The initial marginal load budget is 50 ms.
356 -Hot-path budgets are declared per platform.
357 -Builders may tighten an existing platform budget but may not loosen it.
358 -All thresholds are named constants for later adjustment.
360 -Test runtime does not fail merely for exceeding the five-second target, because a hard short target would reward deleting tests.
361 -Capability mapping remains mandatory.
362 -The watchdog remains a deadlock ceiling.
363 -Marginal load and hot-path budget overruns are warnings because benchmark results require controlled conditions.
365 -Validator-contract version participates in attestation validity.
366 -After a contract update, search marks old attestations outdated and loading requires revalidation.
368 -## Debug logs
370 -Every generated extension writes minimal structured debug logs during normal use.
371 -Builders choose topic-relevant events and enough context to reconstruct branches, state transitions, I/O, timings, and failures.
372 -Logs may include arguments and results when diagnostically useful.
373 -Builders must redact credentials and bound individual values; this is an instruction, not a secret scanner.
375 -Logs use NDJSON with at least:
377 -- timestamp
378 -- extension name and revision
379 -- child session ID
380 -- event name
381 -- duration where applicable
382 -- contextual data
384 -The log path is:
386 -```text
387 -<platform-cache>/pi/ultra/dynamic-extensions/logs/<child-session-id>/<extension-name>.jsonl
388 -```
390 -Ultra derives it from the persistent or in-memory child session ID.
391 -Each extension/child log is capped at 1 MiB.
392 -At the limit, the logger writes one truncation marker and drops later records.
394 -Ultra never ingests logs automatically.
395 -Workflow results expose their paths so the orchestrator can inspect them with normal tools when extension involvement is plausible.
397 -## Runtime observation and repair
399 -Intentional loading is the auditable usage boundary.
400 -Ultra observes successful loading, registrations, dynamic tool calls/results, transcripts, final step results, and debug-log paths.
401 -It cannot externally prove arbitrary hook semantics.
402 -Tests remain the validation proof; runtime logs provide diagnosis.
404 -Validation is revision-wide, not a per-platform load gate.
405 -Search displays tested platforms as advisory metadata only.
406 -An extension validated on another platform may load optimistically on the current platform.
407 -Builders do not preemptively rewrite it solely because current-platform evidence is absent.
409 -When a consumer behaves unexpectedly, Ultra returns extension identities, failure diagnostics, and log paths.
410 -The orchestrator inspects relevant logs, launches a repair builder for the same topic, validates the new canonical revision, and reruns the original workflow.
411 -There is no hard repair-cycle limit.
412 -Runtime failures do not globally mark canonical extensions unhealthy because failures may be task-specific.
414 -Rerunning the original workflow reuses unaffected journal checkpoints.
415 -Canonical names resolve to the repaired hash, changing affected step keys and downstream resolved prompts.
417 -## Reporting
419 -`WorkflowResult` gains a deduplicated `dynamicExtensions` array.
420 -Each entry contains:
422 -- name
423 -- short description
424 -- exact loaded name/hash selector
425 -- exact loaded source path
426 -- associated child debug-log paths
428 -Extensions loaded by successful, failed, or aborted steps are reported after successful binding.
429 -An attempted extension that failed before binding is diagnostic, not used.
430 -Cached/resumed workflow results preserve usage metadata without charging new model usage.
432 -`/ultra run` prints one final dynamic-extension block containing only name, short description, and exact source path.
433 -The model-visible `run_workflow` result contains the complete metadata.
434 -The collapsed tool widget remains compact and shows per-extension information only when expanded.
436 -## Startup and lifecycle
438 -Catalog directories, cleanup, search code, validator code, and generated-extension implementation are lazy-loaded on first dynamic-extension action or first `dynamicExtensions` resolution.
439 -Ordinary Ultra startup and reload perform no catalog scan, process spawn, cleanup, or Git operation.
440 -Reload remains terminal for the invoking handler.
442 -## Acceptance
444 -U008 is complete only when automated tests cover:
446 -- lazy startup with no catalog/process work
447 -- deterministic search ranking and bounded results
448 -- required prior search and duplicate-topic rationale
449 -- create and checkout scaffolds
450 -- manual canonical validation
451 -- all validation gates and structured multi-diagnostics
452 -- validator timeout and abort process-tree cleanup
453 -- source hashing and attestation invalidation
454 -- concurrent publication without clobbering
455 -- stale-base builder recovery
456 -- atomic `current.json` publication on Windows, macOS, and Linux behavior paths
457 -- exact run-local snapshot loading
458 -- additive normal plus dynamic extension loading
459 -- explicit override collision diagnostics
460 -- generated tool activation during `session_start`
461 -- builder-only prompt injection and full tool access without `run_workflow`
462 -- known-name direct consumer loading
463 -- multiple ordered dynamic extensions
464 -- blocked consumer phases after builder failure
465 -- general `writeIsolation` fanout behavior and preserved default read-only law
466 -- dynamic hash participation in step keys and resume behavior
467 -- debug-log format, cap, retention, and result paths
468 -- success, failure, abort, cached, and resumed reporting
469 -- unchanged collapsed tool-widget behavior
470 -- no regression to token/cost accounting
472 -Live acceptance must exercise creation, reuse, a stale or manually invalidated extension, consumer loading, debug-log inspection, repair, rerun checkpoint reuse, parallel isolated builders, cancellation, and final reporting.

extensions/ultra/__tests__/dynamic-extensions.test.ts

Modified
75 75 return result.details;
78 -describe("U008 workflow contract", () => {
78 +describe("dynamic-extension workflow contract", () => {
79 79 it("accepts dynamic selectors and isolated mutating fanout", () => {
80 80 const spec = parseWorkflow({
81 81 name: "builders",
286 286 expect(calls).toEqual(["build"]);
287 287 expect(result.phaseFailures.consume[0].code).toBe(
288 - "u008_builder_failure_blocker",
288 + "dynamic_extension_builder_failure",
290 290 expect(result.phases[1]).toEqual({
291 291 id: "consume",
299 -describe("U008 dynamic catalog", () => {
299 +describe("dynamic-extension catalog", () => {
300 300 it("uses the specified platform cache roots", () => {
301 301 expect(
302 302 dynamicCatalogPaths({

extensions/ultra/__tests__/engine.test.ts

Modified
43 43 function spec(p: {
44 44 name?: string;
45 45 return?: string;
46 + report?: string;
46 47 phases: Phase[];
47 48 }): WorkflowSpec {
48 49 return {
49 50 name: p.name ?? "wf",
50 51 phases: p.phases,
51 52 ...(p.return !== undefined ? { return: p.return } : {}),
53 + ...(p.report !== undefined ? { report: p.report } : {}),
1102 1104 // ---------------------------------------------------------------------------
1104 1106 describe("runWorkflow — result shape", () => {
1107 + it("resolves report independently from return", async () => {
1108 + const { runner } = fakeRunner(() => ({
1109 + ok: true,
1110 + value: { answer: "full", human: ["short"] },
1111 + }));
1112 + const res = await runWorkflow(
1113 + spec({
1114 + return: "{work.results}",
1115 + report: "{work.results[].human[]}",
1116 + phases: [
1118 + id: "work",
1119 + kind: "single",
1120 + step: { summary: "work", prompt: "work" },
1121 + },
1122 + ],
1123 + }),
1124 + undefined,
1125 + baseDeps({ stepRunner: runner }),
1126 + );
1127 + expect(res.result).toEqual([{ answer: "full", human: ["short"] }]);
1128 + expect(res.report).toEqual(["short"]);
1129 + });
1131 + it("normalizes an unresolved report selector to null", async () => {
1132 + const { runner } = fakeRunner(() => ({ ok: true, value: "done" }));
1133 + const res = await runWorkflow(
1134 + spec({
1135 + report: "{args.missing}",
1136 + phases: [
1138 + id: "work",
1139 + kind: "single",
1140 + step: { summary: "work", prompt: "work" },
1141 + },
1142 + ],
1143 + }),
1144 + undefined,
1145 + baseDeps({ stepRunner: runner }),
1146 + );
1147 + expect(res.report).toBeNull();
1148 + });
1105 1150 it("persists every phase's positional results for later inspection", async () => {
1106 1151 const fr = fakeRunner((args) => {
1107 1152 if (args.phase === "review") {

extensions/ultra/__tests__/harness.test.ts

Modified
26 26 } from "@marcfargas/pi-test-harness";
27 27 import { afterEach, describe, expect, it, vi } from "vitest";
28 28 import { DEFAULT_MODEL_TIERS, modelTierPrompt } from "../settings.ts";
29 -import { WorkflowToolSpecSchema } from "../spec.ts";
29 +import { WorkflowSpecSchema } from "../spec.ts";
31 31 let t: TestSession | undefined;
32 32 let root: string | undefined;
172 172 const rules: string[] =
173 173 extension.tools.get("run_workflow").definition.promptGuidelines;
174 174 expect(rules).toEqual([
175 - `Read \`${skillPath}\` once before authoring/editing workflows or calling run_workflow.`,
175 + `read \`${skillPath}\` before authoring or editing workflow specs or calling run_workflow.`,
177 177 expect(t.session.systemPrompt.split(rules[0])).toHaveLength(2);
178 178 expect(t.session.systemPrompt).not.toContain(
251 251 expect(updated).toContain(
252 252 "- fast_extract: custom/extractor; low; Select for literal extraction.",
254 - expect(updated?.match(/# Ultra model tier policy/g)).toHaveLength(1);
254 + expect(updated?.match(/# ultra model tier policy/g)).toHaveLength(1);
255 255 writeFileSync(
256 256 join(t.cwd, ".pi", "settings.json"),
257 257 JSON.stringify({ ultra: { enabled: false } }),
420 420 phaseFailures: {},
421 421 result: [{ status: "blocked", blockers: ["Missing evidence"] }],
422 + report: { summary: "Human report" },
422 423 steered: false,
423 424 dynamicExtensions: [],
424 425 tokenUsage: {
434 435 custom: vi.fn().mockResolvedValue({ status: "completed", result }),
436 437 await extension.commands.get("ultra").handler(`run ${name}`, ctx);
438 + expect(
439 + t.events
440 + .uiCallsFor("notify")
441 + .some(
442 + (call) =>
443 + call.args[0] ===
444 + `ultra: "${name}" complete · 1 phases · $0.0000 · 2 output.`,
445 + ),
446 + ).toBe(true);
437 447 const message = t.session.messages.find(
438 448 (message) =>
439 449 "customType" in message && message.customType === "ultra-result",
443 453 const body = JSON.parse(message.content);
445 455 expect(body.result).toEqual(result.result);
456 + expect(body.report).toBeUndefined();
446 457 expect(body.executionStatus).toBe("finished");
447 458 expect(body.phaseResults).toBeUndefined();
448 459 expect(message.content).not.toContain("INTERMEDIATE");
493 504 expect(t.session.getActiveToolNames()).toContain("run_workflow");
494 505 expect(t.session.systemPrompt).toContain(
495 - "- run_workflow: Run saved or inline multi-agent workflows.",
506 + "- run_workflow: run saved or inline multi-agent workflows.",
497 508 expect(t.session.systemPrompt).toContain(
498 - "once before authoring/editing workflows or calling run_workflow.",
509 + "before authoring or editing workflow specs or calling run_workflow.",
500 511 expect(t.session.messages).toHaveLength(beforeMessages);
501 512 const notifications = t.events.uiCallsFor("notify");
525 536 expect(sendUserMessage).toHaveBeenCalledOnce();
526 537 const prompt = sendUserMessage.mock.calls[0]?.[0];
527 538 expect(prompt).toBe(
528 - "# Ultra exec\n\nFollow run_workflow's authoring skill for the request below; /ultra exec alone does not request a saved workflow.\n\n# User request\n\ninspect this failure once",
539 + "# ultra exec\n\n# user request\n\ninspect this failure once",
530 541 const guidance =
531 542 extension.tools.get("run_workflow").definition.promptGuidelines;
599 610 await extension.commands.get("ultra").handler("run cancelled-run", ctx);
601 612 expect(t.events.uiCallsFor("notify").at(-1)?.args).toEqual([
602 - 'ultra: "cancelled-run" aborted.',
613 + 'ultra: "cancelled-run" aborted · 1 phases · $0.0000 · 0 output.',
603 614 "warning",
605 616 expect(
708 719 phases: [{ id: "work", kind: "single", step: { prompt: "Work" } }],
711 - expected: ["Invalid workflow spec", "/phases/0/step", "summary"],
722 + expected: ["summary"],
714 725 "delivers actionable argument errors to the agent and the tool UI: $params",
783 - it("defers workflow structure while rejecting malformed specs before execution", async () => {
794 + it("exposes the workflow schema while rejecting malformed specs before execution", async () => {
784 795 t = await createTestSession({ extensions: [ultraPath()] });
785 796 const [extension] = t.session.extensionRunner.extensions;
786 797 await extension.commands.get("ultra").getArgumentCompletions("");
788 799 const specSchema = tool.parameters.properties.spec;
789 800 expect(tool.renderCall).toBeTypeOf("function");
790 801 expect(specSchema.type).toBe("object");
791 - expect(specSchema.properties).toEqual({});
792 - expect(specSchema.additionalProperties).toBe(true);
802 + expect(specSchema.properties).toHaveProperty("name");
803 + expect(specSchema.properties).toHaveProperty("phases");
804 + expect(specSchema.properties).toHaveProperty("report");
805 + expect(specSchema.additionalProperties).not.toBe(true);
794 807 expect(tool.description).toContain(
795 - "when parallel work, staged handoffs, or specialist contexts materially help",
808 + "run saved or inline multi-agent workflows.",
797 810 expect(tool.promptSnippet).toBe(
798 - "Run saved or inline multi-agent workflows.",
811 + "run saved or inline multi-agent workflows.",
800 813 expect(tool.promptGuidelines).toHaveLength(1);
801 814 expect(tool.promptGuidelines[0].replaceAll("\\", "/")).toContain(
804 817 expect(tool.parameters.properties.background).toBeUndefined();
805 818 expect(JSON.parse(JSON.stringify(specSchema))).toEqual(
806 - JSON.parse(JSON.stringify(WorkflowToolSpecSchema)),
819 + JSON.parse(JSON.stringify(WorkflowSpecSchema)),
808 821 const step = { summary: "Work", prompt: "Work" };
809 822 for (const spec of [

extensions/ultra/__tests__/interp.test.ts

Modified
106 + it("where supports exact string equality", () => {
107 + const ctx: InterpContext = {
108 + results: {
109 + r: [
110 + { source: "sessions", id: 1 },
111 + { source: "repository", id: 2 },
112 + { source: "sessions", id: 3 },
113 + { source: "a|b", id: 4 },
114 + { source: 7, id: 5 },
115 + { id: 6 },
116 + ],
117 + },
118 + };
119 + expect(
120 + resolveSelector('{r.results | where source == "sessions"}', ctx),
121 + ).toEqual([
122 + { source: "sessions", id: 1 },
123 + { source: "sessions", id: 3 },
124 + ]);
125 + expect(resolveSelector('{r.results | where source == "a|b"}', ctx)).toEqual(
126 + [{ source: "a|b", id: 4 }],
127 + );
128 + });
130 + it("groupBy groups string fields in first-seen order", () => {
131 + const ctx: InterpContext = {
132 + results: {
133 + r: [
134 + { source: "sessions", id: 1 },
135 + { source: "repository", id: 2 },
136 + { source: "sessions", id: 3 },
137 + { source: 7, id: 4 },
138 + { id: 5 },
139 + ],
140 + },
141 + };
142 + expect(resolveSelector("{r.results | groupBy source}", ctx)).toEqual([
144 + key: "sessions",
145 + items: [
146 + { source: "sessions", id: 1 },
147 + { source: "sessions", id: 3 },
148 + ],
149 + },
150 + { key: "repository", items: [{ source: "repository", id: 2 }] },
151 + ]);
152 + });
154 + it("where equality is strict, case-sensitive, and supports empty values", () => {
155 + const ctx: InterpContext = {
156 + results: {
157 + r: [
158 + { source: "sessions", id: 1 },
159 + { source: "Sessions", id: 2 },
160 + { source: "", id: 3 },
161 + { source: false, id: 4 },
162 + { source: 0, id: 5 },
163 + { source: null, id: 6 },
164 + { id: 7 },
165 + ],
166 + },
167 + };
168 + expect(
169 + resolveSelector('{r.results | where source == "Sessions"}', ctx),
170 + ).toEqual([{ source: "Sessions", id: 2 }]);
171 + expect(resolveSelector('{r.results | where source == ""}', ctx)).toEqual([
172 + { source: "", id: 3 },
173 + ]);
174 + expect(resolveSelector('{r.results | where source == "0"}', ctx)).toEqual(
175 + [],
176 + );
177 + });
179 + it("where equality accepts surrounding whitespace", () => {
180 + const ctx: InterpContext = {
181 + results: { r: [{ source: "sessions" }, { source: "repository" }] },
182 + };
183 + expect(
184 + resolveSelector('{r.results | where source == "sessions"}', ctx),
185 + ).toEqual([{ source: "sessions" }]);
186 + });
188 + it("where equality supports escaped strings and pipes", () => {
189 + const ctx: InterpContext = {
190 + results: {
191 + r: [
192 + { source: 'a"b', id: 1 },
193 + { source: "a\\b", id: 2 },
194 + { source: "a|b", id: 3 },
195 + ],
196 + },
197 + };
198 + expect(
199 + resolveSelector(String.raw`{r.results | where source == "a\"b"}`, ctx),
200 + ).toEqual([{ source: 'a"b', id: 1 }]);
201 + expect(
202 + resolveSelector(String.raw`{r.results | where source == "a\\b"}`, ctx),
203 + ).toEqual([{ source: "a\\b", id: 2 }]);
204 + expect(resolveSelector('{r.results | where source == "a|b"}', ctx)).toEqual(
205 + [{ source: "a|b", id: 3 }],
206 + );
207 + });
209 + it("where equality rejects non-string and malformed operands", () => {
210 + const invalid = [
211 + "{r.results | where source == sessions}",
212 + `{r.results | where source != "sessions"}`,
213 + `{r.results | where source == 'sessions'}`,
214 + String.raw`{r.results | where source == "bad\q"}`,
215 + `{r.results | where source == "unterminated}`,
216 + ];
217 + for (const selector of invalid)
218 + expect(() => resolveSelector(selector, fixture())).toThrow(
219 + SpecValidationError,
220 + );
221 + });
223 + it("where and groupBy reject non-array sources and unknown phases", () => {
224 + for (const selector of [
225 + "{item | where real}",
226 + '{item | where real == "yes"}',
227 + "{item | groupBy source}",
228 + "{missing.results | groupBy source}",
229 + ])
230 + expect(() => resolveSelector(selector, fixture())).toThrow(
231 + SpecValidationError,
232 + );
233 + });
235 + it("groupBy handles empty keys and skips invalid keys", () => {
236 + const ctx: InterpContext = {
237 + results: {
238 + r: [
239 + { source: "", id: 1 },
240 + { source: "__proto__", id: 2 },
241 + { source: "", id: 3 },
242 + { source: "__proto__", id: 4 },
243 + { source: null, id: 5 },
244 + { source: false, id: 6 },
245 + { source: ["sessions"], id: 7 },
246 + { id: 8 },
247 + ],
248 + },
249 + };
250 + expect(resolveSelector("{r.results | groupBy source}", ctx)).toEqual([
252 + key: "",
253 + items: [
254 + { source: "", id: 1 },
255 + { source: "", id: 3 },
256 + ],
257 + },
259 + key: "__proto__",
260 + items: [
261 + { source: "__proto__", id: 2 },
262 + { source: "__proto__", id: 4 },
263 + ],
264 + },
265 + ]);
266 + });
268 + it("groupBy returns an empty array for empty or entirely invalid input", () => {
269 + expect(
270 + resolveSelector("{r.results | groupBy source}", { results: { r: [] } }),
271 + ).toEqual([]);
272 + expect(
273 + resolveSelector("{r.results | groupBy source}", {
274 + results: { r: [null, { source: 1 }, { id: 2 }] },
275 + }),
276 + ).toEqual([]);
277 + });
279 + it("groupBy renders as structured JSON in templates", () => {
280 + const output = resolveTemplate("groups={r.results | groupBy source}", {
281 + results: {
282 + r: [
283 + { source: "sessions", id: 1 },
284 + { source: "web", id: 2 },
285 + ],
286 + },
287 + });
288 + expect(output).toBe(
289 + 'groups=[{"key":"sessions","items":[{"source":"sessions","id":1}]},{"key":"web","items":[{"source":"web","id":2}]}]',
290 + );
291 + });
293 + it("rejects chained and repeated selector operators", () => {
294 + for (const selector of [
295 + "{r.results | groupBy source | where key}",
296 + "{r.results | where source | groupBy source}",
297 + ])
298 + expect(() => resolveSelector(selector, fixture())).toThrow(
299 + SpecValidationError,
300 + );
301 + });
106 303 it("resolves and filters {PHASE.failures}", () => {
107 304 expect(resolveSelector("{r.failures}", fixture())).toHaveLength(2);
108 305 expect(
189 386 expect(resolveOver(lit, {})).toEqual(lit);
389 + it("resolves grouped results as a fanout input", () => {
390 + expect(
391 + resolveOver("{r.results | groupBy source}", {
392 + results: {
393 + r: [
394 + { source: "sessions", id: 1 },
395 + { source: "repository", id: 2 },
396 + ],
397 + },
398 + }),
399 + ).toEqual([
400 + { key: "sessions", items: [{ source: "sessions", id: 1 }] },
401 + { key: "repository", items: [{ source: "repository", id: 2 }] },
402 + ]);
403 + });
192 405 it("resolves failures as a recovery fanout", () => {
193 406 expect(resolveOver("{r.failures}", fixture())).toEqual(
194 407 fixture().failures?.r,

extensions/ultra/__tests__/prompt.test.ts

Modified
4 4 import * as fs from "node:fs";
5 5 import { fileURLToPath } from "node:url";
6 6 import { loadSkillsFromDir } from "@earendil-works/pi-coding-agent";
7 7 import { describe, expect, it } from "vitest";
8 8 import { VALIDATOR_CONTRACT_VERSION } from "../dynamic-contract.ts";
9 9 import { resolveSelector, resolveTemplate } from "../interp.ts";
10 -import { DEFAULT_MODEL_TIERS } from "../settings.ts";
11 10 import {
12 11 compileSchema,
13 12 DYNAMIC_EXTENSION_TOOL_NAMES,
14 13 parseWorkflow,
15 - ThinkingLevelSchema,
16 14 WorkflowSpecSchema,
17 - WorkflowToolSpecSchema,
18 15 } from "../spec.ts";
20 17 const PROMPT_PATH = fileURLToPath(
38 35 describe("on-demand workflow authoring skill", () => {
39 - it("keeps a valid skill outside automatic discovery", () => {
36 + it("loads outside automatic discovery and resolves its references", () => {
40 37 const packageJson = JSON.parse(
41 38 fs.readFileSync(
42 39 fileURLToPath(new URL("../package.json", import.meta.url)),
57 54 expect(result.skills.map((skill) => skill.name)).toEqual([
58 55 "ultra-authoring",
60 - const links = [
61 - ...fs.readFileSync(PROMPT_PATH, "utf8").matchAll(/\]\(([^)]+)\)/g),
62 - ];
63 - for (const [, link] of links)
57 + for (const [, link] of fs
58 + .readFileSync(PROMPT_PATH, "utf8")
59 + .matchAll(/\]\(([^)]+)\)/g)) {
64 60 expect(
65 61 fs.existsSync(
66 62 new URL(
71 67 ).toBe(true);
72 - });
74 - it("keeps invocation-critical semantics in prompt assets", () => {
75 - const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
76 - const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
78 - expect(prompt).toContain("# Authoring ultra workflows");
79 - expect(prompt).toContain(
80 - "available on demand, not injected by `/ultra exec`",
81 - );
82 - expect(prompt).not.toContain("canonical spec format is injected below");
83 - expect(prompt).toContain("/ultra run <name>");
84 - expect(prompt).toContain("`run_workflow`");
85 - expect(prompt).toContain("name: ultra-authoring");
86 - expect(reference).toContain("/ultra run <name>");
87 - expect(reference).toContain("Every step returns a JSON object");
88 - expect(prompt).toContain("Ultra refuses to start the workflow");
89 - expect(reference).toContain("Ultra refuses to start the workflow");
90 - expect(prompt).toContain("When confidence is high, act on that choice");
91 - expect(prompt).toContain(
92 - "When confidence is not high, ask one focused question",
93 - );
94 - expect(prompt).toContain(
95 - "Do not treat `/ultra exec` itself or generic creation verbs as evidence for persistence",
96 - );
97 - expect(prompt).not.toContain(
98 - "When in genuine doubt with creation language, persist",
99 - );
100 - const readme = fs.readFileSync(README_PATH, "utf8");
101 - const selectionPolicy =
102 - "independent parallel work, staged agent handoffs, or isolated specialist contexts materially improve the result";
103 - expect(prompt).toContain(selectionPolicy);
104 - expect(readme).toContain(selectionPolicy);
105 - expect(readme).toContain("`/ultra exec <instructions>`");
106 - expect(readme).toContain("`/ultra run <name> [args]`");
107 - expect(readme).toContain("`run_workflow` tool");
108 - expect(readme).not.toContain("`/ultra history`");
109 - expect(readme).not.toContain("`/ultra runs`");
110 - });
112 - it("ties delegation capability and context sufficiency to artifact quality", () => {
113 - const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
114 - const contract = prompt
115 - .split("## Delegation contract\n")[1]
116 - ?.split("\n## ")[0];
117 - expect(contract).toBeDefined();
118 - for (const rule of [
119 - "total tokens and cost of producing an acceptable artifact",
120 - "Never discard binding user instructions",
121 - "Children do not inherit the parent conversation",
122 - "settled decisions and their relevant rationale",
123 - "applicable `AGENTS.md` paths",
124 - "read those instructions before acting",
125 - "provide the tools needed to access them",
126 - "Normal child context-file loading is enabled",
127 - "task-specific parent constraints",
128 - "including nested instructions for the assigned scope",
129 - "prohibitions, required checks, and reporting obligations",
130 - "Available host capability is not authorized scope",
131 - "report as blocked rather than guess",
132 - "effective tier rubric injected into this orchestrator",
133 - "decisions still required, not prompt length or phase position",
134 - "Selecting what matters is a judgment task",
135 - "finish without guessing the user's intent",
136 - "add context, resolve the decision first, or select a more capable tier",
137 - "merge redundant assignments unless independent checks are intentional",
138 - "self-contained `{item}` payloads",
139 - "shared context is justified",
140 - "`maxItems` and `maxLength`",
141 - "preserve blockers and uncertainty",
142 - "uncertainty is not refutation",
143 - "Gate writes on an explicit ready assignment",
144 - "preserve completed, blocked, and unresolved outcomes",
145 - "never infer completion from an empty verification list",
146 - "schema-valid output alone does not establish quality or completion",
147 - ])
148 - expect(contract).toContain(rule);
149 - expect(contract).not.toMatch(/`(?:tiny|small|medium|large|huge)`/);
150 - });
152 - it("documents bounded orchestrator output without intermediate duplication", () => {
153 - for (const path of [PROMPT_PATH, SPEC_FORMAT_PATH, README_PATH]) {
154 - const content = fs.readFileSync(path, "utf8");
155 - for (const term of [
156 - "`phaseResults`",
157 - "UI details",
158 - "50KB",
159 - "2000 lines",
160 - "`fullOutputPath`",
161 - "selectively",
162 - "For both tool and command results",
163 - "full aggregate is always stored once",
164 - "every bounded model-facing envelope includes",
165 - "Complete contents",
166 - "bounded failure diagnostics",
167 - ])
168 - expect(content).toContain(term);
172 - it("retains detailed semantics in the runtime schema", () => {
173 - const toolGuidance = JSON.stringify(WorkflowSpecSchema);
174 - for (const rule of [
71 + it("documents command, child, phase, and result mechanics", () => {
72 + const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
73 + const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
74 + for (const term of [
75 + "/ultra exec <instructions>",
76 + "/ultra run <name> [args]",
77 + "inline spec",
78 + "single",
79 + "fanout",
80 + "children run in separate sessions",
81 + "do not inherit the parent conversation",
82 + "writeIsolation",
83 + "structured_output",
84 + "fullOutputPath",
85 + "thinkingLevel",
86 + ])
87 + expect(prompt).toContain(term);
88 + for (const policy of [
89 + "materially help",
90 + "least-compute",
91 + "token economics",
92 + "Delegation contract",
93 + ])
94 + expect(prompt).not.toContain(policy);
95 + expect(reference).not.toContain("least-compute tier");
96 + const readme = fs.readFileSync(README_PATH, "utf8");
97 + expect(readme).toContain("active `run_workflow` tool advertises");
98 + expect(readme).toContain("`/ultra exec` sends only the user's request");
99 + expect(readme).not.toContain("selection rules live in the authoring skill");
100 + });
102 + it("keeps workflow mechanics synchronized with runtime validation", () => {
103 + const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
104 + const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
105 + const schema = JSON.stringify(WorkflowSpecSchema);
106 + for (const term of [
175 107 "exact key in top-level schemas",
176 108 "{PHASE.results[].FIELD[]}",
109 + '{EXPR | where FIELD == \\"VALUE\\"}',
110 + "{EXPR | groupBy FIELD}",
177 111 "single brace-wrapped selector",
178 112 "Every listed tool must be available",
179 113 "Loaded extensions may activate additional tools",
180 114 "read, grep, find, ls, and bash",
181 115 "forbid mutation",
182 - ]) {
183 - expect(toolGuidance).toContain(rule);
184 - }
185 - });
187 - it("keeps U008 runtime, authoring skill, and builder contract synchronized", () => {
188 - const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
189 - const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
190 - const builder = fs.readFileSync(BUILDER_PROMPT_PATH, "utf8");
191 - const schema = JSON.stringify(WorkflowSpecSchema);
116 + ])
117 + expect(schema).toContain(term);
193 118 for (const tool of DYNAMIC_EXTENSION_TOOL_NAMES) {
194 119 expect(prompt).toContain(tool);
195 120 expect(reference).toContain(tool);
197 122 expect(schema).toContain("dynamicExtensions");
198 123 expect(schema).toContain("writeIsolation");
199 - expect(prompt).toContain("Builder failure blocks later phases");
200 - expect(reference).toContain("u008_builder_failure_blocker");
124 + expect(prompt).toContain("builder failure blocks later phases");
125 + expect(reference).toContain("dynamic_extension_builder_failure");
126 + });
128 + it("keeps the builder contract versioned outside normal workflow guidance", () => {
129 + const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
130 + const builder = fs.readFileSync(BUILDER_PROMPT_PATH, "utf8");
131 + expect(prompt).toContain("dynamic-extension builder contract");
201 132 expect(builder).toContain(
202 133 `dynamic-extension builder contract v${VALIDATOR_CONTRACT_VERSION}`,
204 - for (const field of [
205 - "topic",
206 - "outcome",
207 - "intended consumers",
208 - "done conditions",
209 - ])
210 - expect(prompt).toContain(field);
211 - });
213 - it("makes builder constraints obligations without changing validator version", () => {
214 - const skill = fs.readFileSync(PROMPT_PATH, "utf8");
215 - const builderSection = skill
216 - .split("## Dynamic-extension delegation\n")[1]
217 - ?.split("\n## ")[0];
218 - const builder = fs.readFileSync(BUILDER_PROMPT_PATH, "utf8");
219 - expect(builderSection).toBeDefined();
220 - for (const text of [builderSection, builder]) {
221 - for (const obligation of [
222 - "applicable constraints",
223 - "instruction paths",
224 - "required evidence",
225 - "authority/blocker boundaries",
226 - "obligations, not optional context",
227 - "settled decisions",
228 - "relevant rationale",
229 - "authorized scope",
230 - "available host capability",
231 - "report blocked",
232 - ])
233 - expect(text).toContain(obligation);
234 - }
235 - expect(VALIDATOR_CONTRACT_VERSION).toBe(2);
236 - expect(builder).toMatch(/^# Ultra dynamic-extension builder contract v2\n/);
237 135 expect(builder).toContain("validation gates are unchanged");
238 - expect(builder).not.toContain("Optional brief fields");
239 - expect(builder).not.toContain("full host authority");
136 + expect(prompt).not.toContain("manifest.json");
242 - it("documents isolated child context loading and truthful final outcomes", () => {
243 - const readme = fs.readFileSync(README_PATH, "utf8");
244 - for (const term of [
245 - "Normal child context-file loading is enabled",
246 - "Children do not inherit the parent conversation",
247 - "task-specific parent constraints",
248 - "applicable nested instruction paths",
249 - "Skill and reference contents remain on demand",
250 - "`Plan` readiness gate",
251 - "settled correction brief",
252 - "`completed`, `blocked`, or `unresolved` status",
253 - "explicit unresolved list",
254 - "resolved output schema",
255 - "Reused outputs must satisfy the current schema",
256 - ])
257 - expect(readme).toContain(term);
258 - });
260 - it("requires authored progress summaries beside exact prompts", () => {
261 - const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
262 - const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
263 - const schema = JSON.stringify(WorkflowSpecSchema);
265 - expect(prompt).toContain("every step has a short `summary`");
266 - expect(reference).toContain("exact resolved `prompt`");
267 - expect(schema).toContain("progress-UI sentence");
268 - for (const guidance of [prompt, reference, schema])
269 - expect(guidance).toContain(
270 - "objects as short labels and arrays as counts; prompts retain compact JSON",
271 - );
272 - });
274 - it("keeps the exact interpolation grammar synchronized", () => {
275 - const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
276 - expect(prompt).toContain("{EXPR | where FIELD}");
277 - expect(prompt).not.toContain("{… | where FIELD}");
278 - });
280 - it("keeps phase identifier guidance synchronized", () => {
281 - expect(JSON.stringify(WorkflowSpecSchema)).toContain(
282 - "reserved identifier args",
283 - );
284 - for (const content of [
285 - fs.readFileSync(PROMPT_PATH, "utf8"),
286 - fs.readFileSync(SPEC_FORMAT_PATH, "utf8"),
287 - ]) {
288 - expect(content).toContain("reserved phase ID `args`");
289 - }
290 - });
292 - it("points tier selection at the generated effective rubric", () => {
293 - const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
294 - const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
295 - const readme = fs.readFileSync(README_PATH, "utf8");
297 - expect(prompt).toContain("Assign compute deliberately");
298 - expect(prompt).toContain(
299 - "effective tier rubric injected into this orchestrator",
300 - );
301 - for (const content of [prompt, reference, readme]) {
302 - expect(content).toContain("least-compute tier");
303 - expect(content).toContain("concrete remaining reasoning bottleneck");
304 - expect(content).toContain(
305 - "Broad scope, long prompts, routine review or debugging",
306 - );
307 - expect(content).toMatch(/missing context|Supply missing context/u);
308 - expect(content).toMatch(/report(?:ed)? blocked/u);
309 - }
310 - expect(prompt).toContain("Normally omit `thinkingLevel`");
311 - expect(prompt).toContain(
312 - "Saved workflows retain their explicit model selections",
313 - );
314 - expect(prompt).toContain("within the model's capabilities");
315 - expect(prompt).toContain(
316 - "If every phase uses the same tier, challenge the assignment",
317 - );
318 - expect(prompt).toContain(
319 - "Do not put model-specific instructions in child prompts",
320 - );
321 - expect(reference).toContain("one effective tier rubric");
322 - expect(reference).toContain("normally omit it");
323 - expect(readme).toContain("[`settings.ts`](settings.ts)");
324 - expect(readme).toContain("not in a duplicated fixed tier-role table");
325 - expect(readme).not.toContain("| Selection instruction |");
326 - expect(readme).toContain(
327 - "string aliases and `:thinking` suffixes are not supported",
328 - );
329 - expect(readme).toContain("ordinary tool use and `/ultra exec`");
330 - expect(readme).toContain(
331 - "Saved workflows retain their explicit tier selections",
332 - );
333 - });
335 - it("keeps the opt-in max level in the tier defaults, not child guidance", () => {
336 - expect(ThinkingLevelSchema.enum).toContain("max");
337 - const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
338 - const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
339 - expect(DEFAULT_MODEL_TIERS.huge.thinkingLevel).toBe("max");
340 - expect(reference).toContain("off…max");
341 - expect(prompt).not.toContain("`max`");
342 - });
344 - it("matches runtime positional and structured-output semantics", () => {
345 - const prompt = fs.readFileSync(PROMPT_PATH, "utf8");
346 - const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
348 - expect(reference).toContain(
349 - "last declared parameter absorbs all remaining tokens",
350 - );
351 - expect(reference).not.toContain("extra tokens ignored");
352 - expect(reference).toContain("Every step returns a JSON object");
353 - expect(reference).toContain("`schema` validates");
354 - expect(prompt).toContain(
355 - "how tightly its structured JSON output must be constrained by a schema",
356 - );
357 - });
359 - it("defers workflow structure to the skill without weakening runtime validation", () => {
360 - const wire = JSON.stringify(WorkflowToolSpecSchema);
361 - expect(wire.length).toBeLessThan(150);
362 - expect(WorkflowToolSpecSchema.properties).toEqual({});
363 - const toolValidator = compileSchema(WorkflowToolSpecSchema);
364 - for (const nonObject of [null, [], "workflow", 123])
365 - expect(toolValidator.Check(nonObject)).toBe(false);
366 - const skill = fs.readFileSync(PROMPT_PATH, "utf8");
367 - expect(skill).toContain("read the canonical reference once");
368 - expect(skill).toContain(
369 - "runtime validation still checks the complete workflow",
370 - );
371 - const invalid = {
372 - name: "invalid",
373 - phases: [
374 - {
375 - id: "work",
376 - kind: "fanout",
377 - step: { summary: "Work", prompt: "Work" },
378 - },
379 - ],
380 - };
381 - expect(compileSchema(WorkflowToolSpecSchema).Check(invalid)).toBe(true);
382 - expect(() => parseWorkflow(invalid)).toThrow("Invalid workflow spec");
383 - });
385 - it("keeps canonical review evidence and unique ID joins through unresolved output", () => {
386 - const spec = referenceWorkflow("review");
387 - const [review, verify] = spec.phases;
388 - for (const phase of [review, verify]) {
389 - expect(phase.step.tools).toEqual(["read", "bash", "grep", "find", "ls"]);
390 - for (const term of [
391 - "Original target/base: {args.base}",
392 - "Parent constraints and instruction paths: {args.instructions}",
393 - "AGENTS.md",
394 - "nested instructions",
395 - "read-only Git inspection",
396 - "do not modify anything",
397 - "evidence",
398 - ])
399 - expect(phase.step.prompt).toContain(term);
400 - }
401 - expect(review.step.prompt).toContain(
402 - "stable unique id prefixed by its review dimension",
403 - );
404 - expect(verify.step.prompt).toContain("Echo id, title, and file verbatim");
405 - expect(verify.step.prompt).toContain(
406 - "refuted only with concrete counterevidence",
407 - );
408 - expect(verify.step.prompt).toContain("unresolved when uncertain");
409 - expect(verify.step.prompt).toContain("claim an unrun check passed");
410 - expect(verify.step.prompt).toContain("Finding: {item}");
411 - const schemas = spec.schemas ?? {};
412 - const findings = compileSchema(schemas.Findings);
413 - const verdict = compileSchema(schemas.Verdict);
414 - const candidate = {
415 - id: "correctness:1",
416 - title: "A candidate",
417 - file: "src/route.ts",
418 - detail: "A missing input reaches the unchecked branch.",
419 - evidence: "src/route.ts:12 reads input.value without a guard.",
420 - };
421 - expect(findings.Check({ findings: [candidate] })).toBe(true);
422 - for (const field of ["id", "title", "file", "detail", "evidence"])
423 - expect(
424 - findings.Check({ findings: [{ ...candidate, [field]: undefined }] }),
425 - ).toBe(false);
426 - const judgments = ["confirmed", "refuted", "unresolved"].map(
427 - (status, index) => ({
428 - id: `correctness:${index + 1}`,
429 - title: candidate.title,
430 - file: candidate.file,
431 - status,
432 - why: "Concrete evidence or a stated evidence gap.",
433 - evidence: candidate.evidence,
434 - }),
435 - );
436 - for (const judgment of judgments) {
437 - expect(verdict.Check(judgment)).toBe(true);
438 - for (const field of ["id", "title", "file", "status", "why", "evidence"])
439 - expect(verdict.Check({ ...judgment, [field]: undefined })).toBe(false);
440 - }
441 - expect(verdict.Check({ ...judgments[0], status: "unknown" })).toBe(false);
442 - expect(verdict.Check({ ...judgments[0], why: "" })).toBe(false);
443 - expect(verdict.Check({ ...judgments[0], evidence: "" })).toBe(false);
444 - expect(spec.return).toBe("{verify.results}");
445 - expect(
446 - resolveSelector(spec.return ?? "", { results: { verify: judgments } }),
447 - ).toEqual(judgments);
448 - const bundled = parseWorkflow(
449 - fs.readFileSync(
450 - fileURLToPath(new URL("../workflows/review.json", import.meta.url)),
451 - "utf8",
452 - ),
453 - );
454 - for (const schema of [schemas.Verdict, bundled.schemas?.Verdict])
455 - expect(schema).toMatchObject({
456 - properties: {
457 - id: {
458 - type: "string",
459 - pattern: "^(correctness|security|performance):[A-Za-z0-9_-]+$",
460 - },
461 - },
462 - required: expect.arrayContaining(["id", "status", "why"]),
463 - });
464 - });
466 - it("gives recovery the original read-only assignment, tools, schema, and diagnostics", () => {
467 - const spec = referenceWorkflow("read-package-license");
468 - const [work, recover, report] = spec.phases;
469 - expect(recover.when).toBe("{work.failures}");
470 - expect(recover.over).toBe("{work.failures}");
471 - for (const phase of [work, recover]) {
472 - expect(phase.step.tools).toEqual(["read"]);
473 - expect(phase.step.schema).toBe("License");
474 - for (const term of [
475 - "exact name and license string from the root package.json",
476 - "Scope: that file and applicable instruction files only",
477 - "Read applicable AGENTS.md files before acting",
478 - "Parent constraints and instruction paths: {args.instructions}",
479 - "If that input is missing, return blocked",
480 - "Do not modify anything or run commands",
481 - "cite the path and exact fields in evidence",
482 - "never infer a license",
483 - "Return the License schema",
484 - ])
485 - expect(phase.step.prompt).toContain(term);
486 - }
487 - expect(recover.step.prompt).toContain(
488 - "Failure diagnostics supplement this original assignment: {item}",
489 - );
490 - const failure = {
491 - code: "missing-output",
492 - message: "No submission",
493 - attempts: 1,
494 - };
495 - const resolved = resolveTemplate(recover.step.prompt, {
496 - args: { instructions: "Read AGENTS.md; no network access." },
497 - item: failure,
498 - });
499 - expect(resolved).toContain("root package.json");
500 - expect(resolved).toContain("Read AGENTS.md; no network access.");
501 - expect(resolved).toContain(JSON.stringify(failure));
502 - if (!spec.schemas?.License) throw new Error("Missing License schema");
503 - const schema = compileSchema(spec.schemas.License);
504 - const blocked = {
505 - path: "package.json",
506 - name: "",
507 - license: "",
508 - status: "blocked",
509 - why: "The file could not be read.",
510 - evidence: "read package.json: permission denied",
511 - };
512 - expect(schema.Check(blocked)).toBe(true);
513 - expect(schema.Check({ ...blocked, evidence: undefined })).toBe(false);
514 - expect(schema.Check({ ...blocked, status: "success" })).toBe(false);
515 - expect(report.step.schema).toBe("License");
516 - expect(report.step.prompt).toContain(
517 - "preserving every field including status and evidence",
518 - );
519 - expect(report.step.prompt).toContain(
520 - "Never interpret empty or failed phases as completed extraction",
521 - );
522 - expect(spec.return).toBe("{report.results}");
523 - });
525 - it("keeps the reference doc and validator in lockstep: every complete JSON example parses", () => {
139 + it("keeps the reference doc and validator in lockstep", () => {
526 140 const md = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
527 141 const blocks = [...md.matchAll(/```json\n([\s\S]*?)```/g)].map((m) => m[1]);
528 142 expect(blocks.length).toBeGreaterThan(0);
540 154 !Array.isArray(parsed) &&
541 155 "name" in parsed &&
542 156 "phases" in parsed;
543 - if (isCompleteSpec) {
544 - completeSpecs += 1;
545 - expect(compileSchema(WorkflowToolSpecSchema).Check(parsed)).toBe(true);
546 - const spec = parseWorkflow(parsed);
547 - const context = {
548 - args: { paths: ["README.md"], instructions: "Read AGENTS.md" },
549 - item: "README.md",
550 - results: {} as Record<string, unknown[]>,
551 - failures: {} as Record<string, unknown[]>,
552 - };
553 - for (const phase of spec.phases) {
554 - expect(phase.step.model).toMatch(/^(small|large)$/);
555 - expect(phase.step.thinkingLevel).toBeUndefined();
556 - expect(() =>
557 - resolveTemplate(phase.step.prompt, context),
558 - ).not.toThrow();
559 - expect(() =>
560 - resolveTemplate(phase.step.summary, context),
561 - ).not.toThrow();
562 - for (const selector of [phase.when, phase.over]) {
563 - if (typeof selector === "string")
564 - expect(() => resolveSelector(selector, context)).not.toThrow();
565 - }
566 - context.results[phase.id] = [];
567 - context.failures[phase.id] = [];
157 + if (!isCompleteSpec) continue;
158 + completeSpecs += 1;
159 + expect(compileSchema(WorkflowSpecSchema).Check(parsed)).toBe(true);
160 + const spec = parseWorkflow(parsed);
161 + const context = {
162 + args: { paths: ["README.md"], instructions: "Read AGENTS.md" },
163 + item: "README.md",
164 + results: {} as Record<string, unknown[]>,
165 + failures: {} as Record<string, unknown[]>,
166 + };
167 + for (const phase of spec.phases) {
168 + expect(() => resolveTemplate(phase.step.prompt, context)).not.toThrow();
169 + expect(() =>
170 + resolveTemplate(phase.step.summary, context),
171 + ).not.toThrow();
172 + for (const selector of [phase.when, phase.over]) {
173 + if (typeof selector === "string")
174 + expect(() => resolveSelector(selector, context)).not.toThrow();
569 - const returnSelector = spec.return;
570 - if (returnSelector)
571 - expect(() => resolveSelector(returnSelector, context)).not.toThrow();
176 + context.results[phase.id] = [];
177 + context.failures[phase.id] = [];
179 + for (const selector of [spec.return, spec.report])
180 + if (selector)
181 + expect(() => resolveSelector(selector, context)).not.toThrow();
183 + expect(completeSpecs).toBeGreaterThanOrEqual(20);
184 + });
186 + it("keeps canonical workflow examples addressable", () => {
187 + for (const name of ["review", "read-package-license"]) {
188 + const spec = referenceWorkflow(name);
189 + expect(spec.name).toBe(name);
190 + expect(spec.phases.length).toBeGreaterThan(0);
574 - expect(completeSpecs).toBe(3);

extensions/ultra/__tests__/research.test.ts

Modified
26 26 "report",
28 28 expect(spec.return).toBe("{report.results}");
29 + expect(spec.report).toBe("{report.results}");
31 32 it("compiles every named output schema", () => {
122 123 recommendations: [],
123 124 sources: [],
125 - expect(
126 - resolveSelector(spec.return ?? "", { results: { report: [value] } }),
127 - ).toEqual([value]);
126 + for (const selector of [spec.return, spec.report])
127 + expect(
128 + resolveSelector(selector ?? "", { results: { report: [value] } }),
129 + ).toEqual([value]);
130 132 it("preserves question plus item packets, instruction obligations and explicit gaps", () => {

extensions/ultra/__tests__/settings.test.ts

Modified
224 224 const prompt = modelTierPrompt(tiers);
225 225 expect(prompt).toContain(
226 - "Tier: model; default thinkingLevel; orchestrator-only selection guidance.",
227 - );
228 - expect(prompt).toContain(
229 - "Choose the least-compute tier likely to satisfy each step's acceptance criteria.",
230 - );
231 - expect(prompt).toContain(
232 - "Escalate only for a concrete remaining reasoning bottleneck",
233 - );
234 - expect(prompt).toContain(
235 - "Supply missing context or report blocked instead of compensating with a larger tier.",
226 + "tier: model; default thinkingLevel; orchestrator-only selection guidance.",
228 + for (const instruction of [
229 + "select by residual judgement; default to small; use tiny only when context determines the result.",
230 + "count unresolved decisions/dependencies and downstream consequence.",
231 + "decompose only when it reduces decisions, context, or error propagation more than handoff overhead.",
232 + "credit structured upstream judgement; verify when risk warrants.",
233 + "manage context windows: minimum sufficient context; progressive disclosure.",
234 + "choose least fitting tier; lower wins ties; add verification when one child is insufficient.",
235 + "ask user about missing decisions; encode missing child evidence/authority as a blocker; never guess with higher tier.",
236 + ])
237 + expect(prompt).toContain(instruction);
237 239 for (const [name, tier] of Object.entries(tiers)) {
238 240 expect(prompt).toContain(
239 241 `- ${name}: ${tier.model ?? "session model"}; ${tier.thinkingLevel}; ${tier.instructions}`,

extensions/ultra/__tests__/spec.test.ts

Modified
83 83 describe("parseWorkflow", () => {
84 + it("accepts an independent human report selector", () => {
85 + const spec = { ...validSpec(), report: "{verify.results}" };
86 + expect(parseWorkflow(spec).report).toBe("{verify.results}");
87 + });
84 89 it("parses a valid review-shaped spec (object input)", () => {
85 90 const parsed = parseWorkflow(validSpec());
86 91 expect(parsed.name).toBe("review");

extensions/ultra/__tests__/wire.test.ts

Modified
93 93 result: [{ answer: "FINAL ANSWER" }],
94 + report: { summary: "HUMAN REPORT" },
94 95 steered: false,
95 96 tokenUsage: {
96 97 input: 1,
107 108 const output = await formatWorkflowToolResult(result);
108 109 const body = JSON.parse(output.text);
109 110 expect(body.result).toEqual(result.result);
111 + expect(body.report).toBeUndefined();
110 112 expect(output.text.match(/FINAL ANSWER/g)).toHaveLength(1);
113 + expect(output.text).not.toContain("HUMAN REPORT");
111 114 expect(body.phases).toEqual(result.phases);
112 115 expect(body.tokenUsage).toEqual(result.tokenUsage);
113 116 expect(body.steered).toBe(false);
134 137 expect(JSON.parse(await readFile(output.fullOutputPath, "utf8"))).toEqual(
135 138 result,
140 + expect(
141 + JSON.parse(await readFile(output.fullOutputPath, "utf8")).report,
142 + ).toEqual(result.report);
137 143 expect(result.phaseFailures.review[0].transcriptTail).toBe(
138 144 "PRIVATE TRANSCRIPT",
207 213 .map(stripVTControlCharacters);
216 + it("renders only the configured report in collapsed and expanded views", () => {
217 + const markdown = formatUltraResultMarkdown({
218 + workflow: "custom",
219 + phases: [{ id: "work", ran: 1, ok: 1, dropped: 0 }],
220 + phaseResults: {},
221 + phaseFailures: {},
222 + result: [{ secret: "AGENT ONLY" }],
223 + report: { summary: "HUMAN REPORT", findings: ["one"] },
224 + steered: false,
225 + tokenUsage: {
226 + input: 10,
227 + output: 20,
228 + total: 30,
229 + cacheRead: 0,
230 + cacheWrite: 0,
231 + cost: 0.4,
232 + },
233 + });
234 + expect(markdown).toContain("HUMAN REPORT");
235 + expect(markdown).not.toContain("AGENT ONLY");
236 + expect(markdown).not.toContain("Usage:");
237 + for (const expanded of [false, true]) {
238 + const output = formatUltraResultMarkdown(
240 + workflow: "custom",
241 + phases: [{ id: "work", ran: 1, ok: 1, dropped: 0 }],
242 + phaseResults: {},
243 + phaseFailures: {},
244 + result: [{ secret: "AGENT ONLY" }],
245 + report: { summary: "HUMAN REPORT", findings: ["one"] },
246 + steered: false,
247 + tokenUsage: {
248 + input: 10,
249 + output: 20,
250 + total: 30,
251 + cacheRead: 0,
252 + cacheWrite: 0,
253 + cost: 0.4,
254 + },
255 + },
256 + expanded,
257 + );
258 + expect(output).toContain("HUMAN REPORT");
259 + expect(output).not.toContain("AGENT ONLY");
260 + expect(output).not.toContain("Usage:");
261 + expect(output).not.toContain("Workflow details");
263 + });
210 265 it("renders compact drops and expanded typed diagnostics", () => {
211 266 const failure = {
212 267 code: "transport" as const,

extensions/ultra/dynamic-validator-worker.ts

Modified
193 193 actual: error instanceof Error ? error.message : String(error),
194 194 remediation:
195 - "Write valid manifest.json using the U008 manifest contract.",
195 + "Write valid manifest.json using the dynamic-extension builder contract.",
197 197 return;

extensions/ultra/engine.ts

Modified
2 2 // ultra — phase engine (Task 6).
4 4 // Sequences a workflow's phases, owns the interpolation mini-syntax (it alone
5 5 // holds `args` + prior-phase `results`), fans each phase out through the
6 -// concurrency pool, normalises drops/aborts, resolves the `return` selector, and
7 -// produces the documented aggregate `{ workflow, phases, phaseResults, phaseFailures, result, steered, tokenUsage }`.
6 +// concurrency pool, normalises drops/aborts, resolves the `return`/`report`
7 +// selectors, and produces the documented aggregate `{ workflow, phases, phaseResults, phaseFailures, result, report, steered, tokenUsage }`.
9 9 // Everything model/network-touching is behind the injected `stepRunner` (Task
10 10 // 5's `StepRunner` shape), so this module is fully offline-testable with a fake
75 75 * (which fans `session.abort()` to each live session — the signal never
76 76 * reaches a session itself, validated P1). */
77 77 signal?: AbortSignal;
78 - /** Lazy U008 catalog/tool support, present only for workflows that request it. */
78 + /** Lazy dynamic-extension catalog/tool support, present only for workflows that request it. */
79 79 dynamicExtensions?: DynamicExtensionResolver;
80 80 /** Live progress sink (overlay or tool surface); absent ⇒ headless no-op. */
81 81 onProgress?: (event: UltraProgressEvent) => void;
118 118 /** Failed steps enriched with phase position and original fanout item. */
119 119 phaseFailures: Record<string, PhaseFailure[]>;
120 120 result: unknown;
121 + /** Optional human-facing report selected independently from `result`. */
122 + report?: unknown;
121 123 steered: boolean;
122 124 tokenUsage: TokenUsage;
123 125 /** Exact validated dynamic-extension revisions successfully bound by children. */
353 355 agentIds,
355 357 // Ordinary all-dropped phases remain visible and continue. A builder drop is
356 - // different: loading unbuilt or stale capability later would violate U008's
358 + // different: loading unbuilt or stale capability later would violate the dynamic-extension
357 359 // trust boundary, so every later phase becomes a typed blocked outcome.
358 360 if (isDynamicExtensionBuilderStep(phase.step) && dropped > 0) {
359 361 blockRemainingPhases(
371 373 const lastPhase = spec.phases[spec.phases.length - 1];
374 + const interpolation = { args, results, failures: phaseFailures };
372 375 const result =
373 376 spec.return !== undefined
374 - ? resolveSelector(spec.return, {
375 - args,
376 - results,
377 - failures: phaseFailures,
378 - })
377 + ? resolveSelector(spec.return, interpolation)
379 378 : results[lastPhase.id];
379 + const report =
380 + spec.report !== undefined
381 + ? (resolveSelector(spec.report, interpolation) ?? null)
382 + : undefined;
381 384 if (!deps.signal?.aborted && phases.every((phase) => phase.dropped === 0)) {
382 385 await deps.journal?.complete?.();
390 393 phaseResults,
391 394 phaseFailures,
392 395 result,
396 + ...(spec.report !== undefined ? { report } : {}),
393 397 steered: steeredBox.value,
394 398 tokenUsage,
395 399 dynamicExtensions: collectDynamicExtensions(phaseResults),
538 542 ? phase.over[index]
539 543 : undefined;
540 544 const failure: StepFailure = {
541 - code: "u008_builder_failure_blocker",
545 + code: "dynamic_extension_builder_failure",
542 546 message: "blocked because a dynamic-extension builder failed",
543 547 retryable: true,
544 548 attempts: 0,

extensions/ultra/implementation.ts

Modified
16 16 import { Type } from "typebox";
17 17 import { createActionSummarizer } from "./action-summarizer.ts";
18 18 import { renderUltraCall } from "./call.ts";
19 +import { formatUsage } from "./display.ts";
19 20 import {
20 21 BUILTIN_TOOL_NAMES,
21 22 makeResolveModel,
45 46 discoverWorkflows,
46 47 missingWorkflowTools,
47 48 type WorkflowSpec,
48 - WorkflowToolSpecSchema,
49 + WorkflowSpecSchema,
49 50 workflowUsesDynamicExtensions,
50 51 } from "./spec.ts";
51 52 import { renderUltraResult, UltraOverlayComponent } from "./ui.ts";
69 70 return process.platform === "win32" ? absolute.toLowerCase() : absolute;
73 +function notifyWorkflowCompletion(
74 + ctx: ExtensionContext,
75 + result: WorkflowResult,
76 +): void {
77 + if (!ctx.hasUI) return;
78 + const dropped = result.phases.reduce(
79 + (count, phase) => count + phase.dropped,
80 + 0,
81 + );
82 + const status = result.aborted
83 + ? "aborted"
84 + : dropped > 0
85 + ? "incomplete"
86 + : "complete";
87 + const usage = formatUsage(result.tokenUsage).summary;
88 + ctx.ui.notify(
89 + `ultra: "${result.workflow}" ${status} · ${result.phases.length} phases · ${usage}.`,
90 + status === "complete" ? "info" : "warning",
91 + );
72 94 export function resolveSubagentExtensions(
73 95 setting: UltraSettings["subagentExtensions"],
74 96 ): SubagentExtensionSelection {
262 284 name: "run_workflow",
263 285 label: "run_workflow",
264 286 description:
265 - "Run a saved or inline multi-agent workflow when parallel work, staged handoffs, or specialist contexts materially help. " +
266 - "Returns selected output, failures, usage, and fullOutputPath for the complete evidence; results above 50KB or 2000 lines return a bounded file reference.",
267 - promptSnippet: "Run saved or inline multi-agent workflows.",
287 + "run saved or inline multi-agent workflows. " +
288 + "return selected output, failures, usage, and fullOutputPath for complete evidence; results above 50KB or 2000 lines return a bounded file reference.",
289 + promptSnippet: "run saved or inline multi-agent workflows.",
268 290 promptGuidelines: [
269 - `Read \`${fileURLToPath(new URL("./prompts/ultra-authoring/SKILL.md", import.meta.url))}\` once before authoring/editing workflows or calling run_workflow.`,
291 + `read \`${fileURLToPath(new URL("./prompts/ultra-authoring/SKILL.md", import.meta.url))}\` before authoring or editing workflow specs or calling run_workflow.`,
271 293 parameters: Type.Object({
272 294 name: Type.Optional(
274 296 description: "Saved workflow name (project/global/bundled).",
277 - spec: Type.Optional(WorkflowToolSpecSchema),
299 + spec: Type.Optional(WorkflowSpecSchema),
278 300 args: Type.Optional(
279 301 Type.Record(Type.String(), Type.Unknown(), {
280 302 description: "Workflow inputs, available as {args.NAME}.",
330 352 onProgress: progress.sink,
332 354 const output = await formatWorkflowToolResult(result);
355 + notifyWorkflowCompletion(ctx, result);
333 356 return {
334 357 content: [{ type: "text", text: output.text }],
335 358 details: {
349 372 const sendExecPrompt = (request: string) => {
350 373 activateWorkflowTool();
351 - pi.sendUserMessage(
352 - `# Ultra exec\n\nFollow run_workflow's authoring skill for the request below; /ultra exec alone does not request a saved workflow.\n\n# User request\n\n${request}`,
353 - );
374 + pi.sendUserMessage(`# ultra exec\n\n# user request\n\n${request}`);
356 377 // ── `/ultra [exec | run]` command ──
518 539 ctx.ui.notify(`ultra: "${spec.name}" aborted.`, "warning");
519 540 return;
521 - const count = Array.isArray(result.result)
522 - ? `${result.result.length} results`
523 - : "done";
524 542 const output = await formatWorkflowToolResult(result);
525 543 pi.sendMessage({
526 544 customType: "ultra-result",
528 546 display: true,
529 547 details: { ...result, fullOutputPath: output.fullOutputPath },
531 - if (outcome.status === "aborted")
532 - ctx.ui.notify(`ultra: "${spec.name}" aborted.`, "warning");
533 - else
534 - ctx.ui.notify(
535 - `ultra: "${result.workflow}" complete — ${result.phases.length} phases, ${count}` +
536 - `${result.steered ? " (steered)" : ""}.`,
537 - "info",
538 - );
549 + notifyWorkflowCompletion(ctx, result);

extensions/ultra/interp.ts

Modified
9 9 // {item} — current fanout item (string verbatim; else compact JSON)
10 10 // {PHASE.results} — a prior phase's results array (nulls RETAINED positionally)
11 11 // {PHASE.failures} — a prior phase's typed failure array
12 -// {PHASE.results[].FIELD[]} — flatten: concat each result's FIELD array (null elems / missing FIELDs SKIPPED)
13 -// {EXPR | where FIELD} — keep elements whose boolean FIELD is truthy (null elems / missing FIELD SKIPPED)
12 +// {PHASE.results[].FIELD[]} — flatten: concat each result's FIELD array (null elems / missing FIELDs SKIPPED)
13 +// {EXPR | where FIELD} — keep elements whose boolean FIELD is truthy (null elems / missing FIELD SKIPPED)
14 +// {EXPR | where FIELD == "VALUE"} — keep elements whose string FIELD exactly equals VALUE
15 +// {EXPR | groupBy FIELD} — group records by exact string FIELD value
15 17 // Pure module: no SDK imports.
43 45 const RE_FLATTEN = new RegExp(
44 46 `^(${IDENT})\\.results\\[\\]\\.(${IDENT})\\[\\]$`,
46 -const RE_WHERE = new RegExp(`^where\\s+(${IDENT})$`);
47 -// A selector value (`over`/`return`) is exactly one braced token, no nesting.
48 +const JSON_STRING = String.raw`"(?:\\.|[^"\\])*"`;
49 +const RE_WHERE = new RegExp(
50 + `^where\\s+(${IDENT})(?:\\s*==\\s*(${JSON_STRING}))?$`,
51 +);
52 +const RE_GROUP_BY = new RegExp(`^groupBy\\s+(${IDENT})$`);
53 +// A selector value (`over`/`when`/`return`/`report`) is exactly one braced token, no nesting.
48 54 const RE_SINGLE_TOKEN = /^\{([^{}]*)\}$/;
49 55 // Template token scanner: every `{...}` (no nesting in v1).
50 56 const RE_TOKEN = /\{([^{}]*)\}/g;
56 62 | { kind: "failures"; phase: string }
57 63 | { kind: "flatten"; phase: string; field: string };
59 -type Expr = Base | { kind: "where"; base: Base; field: string };
65 +type Expr =
66 + | Base
67 + | { kind: "where"; base: Base; field: string; equals?: string }
68 + | { kind: "groupBy"; base: Base; field: string };
61 70 function isRecord(value: unknown): value is Record<string, unknown> {
62 71 return typeof value === "object" && value !== null && !Array.isArray(value);
83 92 throw new SpecValidationError(
84 - `unknown interpolation token "{${raw}}" (outside the v1 grammar; allowed: args.NAME, item, PHASE.results, PHASE.failures, PHASE.results[].FIELD[])`,
93 + `unknown interpolation token "{${raw}}" (outside the v1 grammar; allowed: args.NAME, item, PHASE.results, PHASE.failures, PHASE.results[].FIELD[], and selector operators where/groupBy)`,
97 +function selectorPipeIndex(input: string, start = 0): number {
98 + let quoted = false;
99 + let escaped = false;
100 + for (let index = start; index < input.length; index++) {
101 + const char = input[index];
102 + if (quoted) {
103 + if (escaped) escaped = false;
104 + else if (char === "\\") escaped = true;
105 + else if (char === '"') quoted = false;
106 + } else if (char === '"') quoted = true;
107 + else if (char === "|") return index;
109 + return -1;
88 112 function parseExpr(inner: string): Expr {
89 113 const trimmed = inner.trim();
90 114 if (trimmed === "") {
91 115 throw new SpecValidationError('empty interpolation token "{}"');
94 - const pipeIdx = trimmed.indexOf("|");
118 + const pipeIdx = selectorPipeIndex(trimmed);
95 119 if (pipeIdx === -1) return parseBase(trimmed);
97 121 const left = trimmed.slice(0, pipeIdx).trim();
98 122 const right = trimmed.slice(pipeIdx + 1).trim();
99 - if (right.includes("|")) {
123 + if (selectorPipeIndex(trimmed, pipeIdx + 1) !== -1) {
100 124 throw new SpecValidationError(
101 - `only a single "| where FIELD" filter is allowed in "{${trimmed}}"`,
125 + `only a single selector operator is allowed in "{${trimmed}}"`,
128 + const base = parseBase(left);
129 + const groupBy = right.match(RE_GROUP_BY);
130 + if (groupBy) return { kind: "groupBy", base, field: groupBy[1] };
104 132 const filter = right.match(RE_WHERE);
105 - if (!filter) {
106 - throw new SpecValidationError(
107 - `unsupported filter "${right}" in "{${trimmed}}" (only "| where FIELD" is allowed)`,
108 - );
133 + if (filter) {
134 + let equals: string | undefined;
135 + if (filter[2] !== undefined) {
136 + try {
137 + equals = JSON.parse(filter[2]) as string;
138 + } catch {
139 + throw new SpecValidationError(
140 + `where equality requires a valid JSON string in "{${trimmed}}"`,
141 + );
144 + return {
145 + kind: "where",
146 + base,
147 + field: filter[1],
148 + ...(equals !== undefined ? { equals } : {}),
149 + };
110 - return { kind: "where", base: parseBase(left), field: filter[1] };
151 + throw new SpecValidationError(
152 + `unsupported selector operator "${right}" in "{${trimmed}}" (use "| where FIELD", '| where FIELD == "VALUE"', or "| groupBy FIELD")`,
153 + );
113 156 function getResults(ctx: InterpContext, phase: string): unknown[] {
173 216 for (const el of source) {
174 217 if (el === null || el === undefined) continue; // skip null elements
175 218 if (!isRecord(el)) continue; // missing FIELD (not dereferenceable)
176 - if (el[expr.field]) out.push(el); // keep truthy FIELD
219 + const value = el[expr.field];
220 + if (
221 + expr.equals !== undefined
222 + ? typeof value === "string" && value === expr.equals
223 + : Boolean(value)
225 + out.push(el);
178 227 return out;
229 + if (expr.kind === "groupBy") {
230 + const source = resolveBase(expr.base, ctx);
231 + if (!Array.isArray(source)) {
232 + throw new SpecValidationError(
233 + `"| groupBy ${expr.field}" requires an array on its left-hand side`,
234 + );
236 + const groups: Array<{ key: string; items: unknown[] }> = [];
237 + const byKey = new Map<string, { key: string; items: unknown[] }>();
238 + for (const el of source) {
239 + if (!isRecord(el)) continue;
240 + const value = el[expr.field];
241 + if (typeof value !== "string") continue;
242 + const group = byKey.get(value);
243 + if (group) {
244 + group.items.push(el);
245 + continue;
247 + const created = { key: value, items: [el] };
248 + byKey.set(value, created);
249 + groups.push(created);
251 + return groups;
180 253 return resolveBase(expr, ctx);
206 - * Resolve a single braced selector expression (`over` / `return`) to its raw
279 + * Resolve a single braced selector expression (`over` / `when` / `return` /
280 + * `report`) to its raw
207 281 * value. The whole input must be exactly one `{...}` token.
209 283 export function resolveSelector(selector: string, ctx: InterpContext): unknown {

extensions/ultra/prompts/dynamic-extension-builder.md

Modified
1 -# Ultra dynamic-extension builder contract v2
1 +# dynamic-extension builder contract v2
3 -You own exactly one purpose-built Pi extension topic.
4 -The extension has the orchestrator's available host capability and is not sandboxed.
5 -Available host capability is not authorized scope: act only within the brief's authority and applicable user instructions.
6 -Search before deciding whether to reuse, checkout, improve, repair, or create.
7 -Read only relevant candidate READMEs, then use normal tools on the returned draft path.
8 -Call `validate_dynamic_extension` and do not report success until it publishes a validated revision.
9 -A stale-base result requires checking out current canonical state, reapplying intent, and validating again.
3 +- own exactly one purpose-built Pi extension topic.
4 +- use the orchestrator's available host capability only within the brief's authority and applicable user instructions.
5 +- search before deciding whether to reuse, checkout, improve, repair, or create.
6 +- read only relevant candidate READMEs, then use normal tools on the returned draft path.
7 +- call `validate_dynamic_extension`; report success only after it publishes a validated revision.
8 +- after a stale-base result, checkout current canonical state, reapply intent, and validate again.
11 -Your brief must identify topic, outcome, intended consumers, done conditions, applicable constraints, instruction paths, required evidence, and authority/blocker boundaries.
12 -These are obligations, not optional context; an explicit statement that none apply is different from missing information.
13 -Read applicable instructions, including nested `AGENTS.md` files for the assigned scope, and required evidence before acting.
14 -Preserve binding user instructions, settled decisions, and their relevant rationale throughout implementation and validation.
15 -Use assumptions and platform facts only when supported; report blocked instead of guessing missing intent, evidence, or authority.
16 -Children do not inherit the parent conversation; inspect relevant sources yourself instead of requesting pasted transcripts or general Pi documentation.
17 -Normal context-file loading does not replace task-specific parent constraints and nested instruction paths in the brief.
18 -This contract changes authoring obligations only; validator contract version 2, manifest schema, and validation gates are unchanged.
10 +## brief requirements
20 -Generated extensions obey these laws:
12 +- identify the topic, outcome, intended consumers, done conditions, applicable constraints, instruction paths, required evidence, and authority/blocker boundaries.
13 +- treat these as obligations, not optional context; explicitly state when none apply.
14 +- read applicable instructions, including nested `AGENTS.md` files for the assigned scope, and required evidence before acting.
15 +- preserve binding user instructions, settled decisions, and their relevant rationale through implementation and validation.
16 +- use assumptions and platform facts only when supported.
17 +- report blocked instead of guessing missing intent, evidence, or authority.
18 +- inspect relevant sources yourself; children do not inherit the parent conversation.
19 +- do not request pasted transcripts or general Pi documentation.
20 +- normal context-file loading does not replace task-specific parent constraints and nested instruction paths in the brief.
21 +- this contract changes authoring obligations only; validator contract version 2, manifest schema, and validation gates are unchanged.
22 -- Cover one durable topic; modify an existing extension while responsibility remains in that topic.
23 -- Use a two-to-four-term kebab-case domain capability name without platform, version, model, session, helper, or utils terms.
24 -- Keep `manifest.json` at schema version 1 with `name`, `description`, `topic`, `scope`, `nonGoals`, `tags`, `capabilities`, `requiredExecutables`, `optionalExecutables`, `overrides`, and per-platform `benchmarkBudgets` containing `hotPathMs`.
25 -- Keep Purpose, Scope, Non-goals, Alternatives considered, Decision rationale, Usage, Architecture, Test matrix, and Promotion notes in README.md.
26 -- Map every capability ID into the README test matrix and an exact `node:test` case name.
27 -- Keep `index.ts` thin and startup non-blocking; defer optional work and never probe subprocesses or broadly scan at startup.
28 -- Activate every generated tool during `session_start`; support headless `print` mode and guard TUI-only behavior.
29 -- Propagate aborts, bound tool output, render compactly by default with expanded detail, and return typed actionable errors.
30 -- Prefer native Node APIs over shell helpers; use erasable TypeScript supported by Node 22.19+.
31 -- Add no third-party runtime npm dependencies; declare imported Pi host packages as peers only.
32 -- Check declared executables through native PATH access, warning for missing required tools and informing for missing optional tools without subprocess probes.
33 -- Use local fakes or loopback services in fast tests; never use external networks.
34 -- Keep `debug.ts` byte-identical to the scaffold and emit bounded credential-redacted topic events with enough context to reconstruct decisions, I/O, timings, and failures.
35 -- Never intentionally modify loaded catalog source; runtime state belongs in session entries or explicit project/user data paths.
36 -- Declare every intentional collision as an exact qualified override such as `tool:read`.
37 -- Never register or override Ultra's reserved `run_workflow`, `structured_output`, or four dynamic-extension builder tool names.
38 -- Preserve or tighten existing platform hot-path budgets; never loosen them.
39 -- Treat load and hot-path budget warnings as benchmark evidence to reproduce under controlled conditions, not correctness failures.
40 -- Keep Markdown prose semantic, one sentence per line.
23 +## generated extensions
42 -Validation runs all independent correctness gates plus advisory performance checks, retains failed or aborted drafts, and publishes only when correctness succeeds.
25 +- cover one durable topic; modify an existing extension only while responsibility remains in that topic.
26 +- use a two-to-four-term kebab-case domain capability name without platform, version, model, session, helper, or utils terms.
27 +- keep `manifest.json` at schema version 1 with `name`, `description`, `topic`, `scope`, `nonGoals`, `tags`, `capabilities`, `requiredExecutables`, `optionalExecutables`, `overrides`, and per-platform `benchmarkBudgets` containing `hotPathMs`.
28 +- keep Purpose, Scope, Non-goals, Alternatives considered, Decision rationale, Usage, Architecture, Test matrix, and Promotion notes in `README.md`.
29 +- map every capability ID into the README test matrix and an exact `node:test` case name.
30 +- keep `index.ts` thin and startup non-blocking; defer optional work and never probe subprocesses or broadly scan at startup.
31 +- activate every generated tool during `session_start`; support headless `print` mode and guard TUI-only behavior.
32 +- propagate aborts, bound tool output, render compactly by default with expanded detail, and return typed actionable errors.
33 +- prefer native Node APIs over shell helpers; use erasable TypeScript supported by Node 22.19+.
34 +- add no third-party runtime npm dependencies; declare imported Pi host packages as peers only.
35 +- check declared executables through native PATH access, warning for missing required tools and informing for missing optional tools without subprocess probes.
36 +- use local fakes or loopback services in fast tests; never use external networks.
37 +- keep `debug.ts` byte-identical to the scaffold and emit bounded credential-redacted topic events with enough context to reconstruct decisions, I/O, timings, and failures.
38 +- never intentionally modify loaded catalog source; runtime state belongs in session entries or explicit project/user data paths.
39 +- declare every intentional collision as an exact qualified override such as `tool:read`.
40 +- never register or override Ultra's reserved `run_workflow`, `structured_output`, or four dynamic-extension builder tool names.
41 +- preserve or tighten existing platform hot-path budgets; never loosen them.
42 +- treat load and hot-path budget warnings as benchmark evidence to reproduce under controlled conditions, not correctness failures.
43 +- keep Markdown prose semantic, one sentence per line.
45 +## validation
47 +- run all independent correctness gates plus advisory performance checks.
48 +- retain failed or aborted drafts.
49 +- publish only when correctness succeeds.

extensions/ultra/prompts/references/spec-format.md

Modified
1 1 # ultra workflow spec — canonical reference
3 -A workflow is a single JSON object. `parseWorkflow` (the same validator the
4 -runtime uses) checks it structurally and rejects anything malformed. This file
5 -is the authoritative format reference; the extension `README.md` links here
6 -rather than restating it.
3 +- a workflow is a single JSON object.
4 +- `parseWorkflow` uses the runtime validator to check it structurally and reject malformed specs.
5 +- this file is the authoritative format reference; the extension `README.md` links here rather than restating it.
8 -## Top-level fields
7 +## machine-readable contract
10 -| Field | Required | Type | Meaning |
11 -| ------------- | -------- | --------------------------- | -------------------------------------------------------------- |
12 -| `name` | yes | string (non-empty) | Workflow id. The file is invoked as `/ultra run <name>`. |
13 -| `description` | no | string | One line shown in the picker. |
14 -| `args` | no | object (`{ hint?, ... }`) | Declares run inputs; `hint` is the usage shown after the name. |
15 -| `schemas` | no | record<string, JSON Schema> | Named, reusable output schemas referenced by steps. |
16 -| `phases` | yes | array (≥1) of phases | The ordered stages. Each is `single` or `fanout`. |
17 -| `return` | no | string (one selector token) | Selects the workflow's final value from phase results. |
9 +- the active `run_workflow` tool exposes the complete `WorkflowSpecSchema` for inline `spec` values.
10 +- that schema is the source for field names, requiredness, types, unions, and field descriptions.
11 +- `parseWorkflow()` remains the source for semantic checks and applies to saved and inline specs alike.
12 +- this reference avoids repeating the structural schema; it documents execution semantics, authoring rules, and valid examples.
19 -`args` may also declare `params`: an ordered list of input names. A bare
20 -`/ultra run <name> tokA tokB` maps tokens onto them positionally; the last declared parameter absorbs all remaining tokens so it can carry free text. Values are exposed as `{args.NAME}`. Without `params`, pass inputs as a trailing JSON object: `/ultra run <name> {"k": "v"}`. A missing `{args.NAME}` renders to the empty string.
14 +- `args` may declare `params`, an ordered list of input names.
15 +- `/ultra run <name> tokA tokB` maps tokens positionally; the last declared parameter absorbs remaining tokens as free text.
16 +- values are exposed as `{args.NAME}`.
17 +- without `params`, pass inputs as a trailing JSON object: `/ultra run <name> {"k": "v"}`.
18 +- a missing `{args.NAME}` renders to the empty string.
22 -## Orchestrator output
20 +## orchestrator output
24 -`return` selects the final value; without it, the last phase's results are used.
25 -For both tool and command results, the full aggregate is always stored once and every bounded model-facing envelope includes `fullOutputPath`.
26 -The selected value appears once when it fits, with phase counts, bounded failure diagnostics, and usage, not duplicated `phaseResults`.
27 -`executionStatus` is `finished`, `incomplete` for dropped steps, or `aborted`; `finished` describes execution only, not artifact quality or semantic success.
28 -Oversized values use a retrieval notice within the 50KB or 2000 lines limit.
29 -Complete contents, including intermediate results and failure inputs/transcript tails, remain in UI details.
30 -Read that file selectively when needed, rather than loading all intermediate work into context.
22 +- `return` selects the final value; without it, the last phase's results are used.
23 +- `report` optionally selects the human-facing value; without it, human rendering uses the existing `return`-based formatting.
24 +- tool and command results store the full aggregate once; every bounded model-facing envelope includes `fullOutputPath`.
25 +- the selected `return` value appears once in the bounded model-facing envelope when it fits, with phase counts, bounded failure diagnostics, and usage; `phaseResults` is not duplicated.
26 +- the selected `report` value is persisted in the full aggregate and rendered only on human-facing surfaces; it is omitted from the bounded model-facing envelope.
27 +- `executionStatus` is `finished`, `incomplete` for dropped steps, or `aborted`.
28 +- `finished` describes execution only, not artifact quality or semantic success.
29 +- oversized values use a retrieval notice within the 50KB or 2000 lines limit.
30 +- complete contents, including intermediate results and failure inputs/transcript tails, remain in UI details.
31 +- read that file selectively when needed instead of loading all intermediate work into context.
32 -## Phases
33 +## phases
34 -A phase has an `id` (referenced by interpolation and `return`), a `kind`, a
35 -`step`, an optional `concurrency` cap, an optional `when` selector, and optional fan-out `writeIsolation`.
36 -Phase IDs must be unique identifiers matching `[A-Za-z_][A-Za-z0-9_]*`; the reserved phase ID `args` is forbidden because it belongs to the workflow-input namespace.
35 +- a phase has an `id`, `kind`, `step`, optional `concurrency`, optional `when`, and optional fan-out `writeIsolation`.
36 +- `id` is referenced by interpolation, `return`, and `report`.
37 +- phase IDs must match `[A-Za-z_][A-Za-z0-9_]*` and be unique.
38 +- the reserved phase ID `args` is forbidden because it belongs to the workflow-input namespace.
38 -- **`single`** — runs the step exactly once, for one synthesis / judge /
39 - aggregation pass over prior results. (`over` is accepted by the schema but
40 - **ignored** in v1; for per-item parallelism use `fanout`.)
41 -- **`fanout`** — runs the step once per element of `over`, concurrently (capped
42 - by `concurrency` and the engine's global pool). `over` is **required**.
40 +- **`single`** — runs the step once for synthesis, judging, or aggregation over prior results; `over` is accepted but ignored in v1.
41 +- **`fanout`** — runs the step once per `over` element, concurrently within `concurrency` and the engine's global pool; `over` is required.
44 -`when` uses the existing single-selector grammar.
45 -If it resolves to `false`, `null`, `undefined`, an empty string, or an empty array, the phase is skipped and records `{ ran: 0, ok: 0, dropped: 0 }` with empty results.
46 -Any other value runs the phase.
47 -A skipped phase may still be referenced later through empty `{PHASE.results}` and `{PHASE.failures}` arrays.
48 -There are no loops or arbitrary expressions.
43 +- `when` uses the existing single-selector grammar.
44 +- `false`, `null`, `undefined`, an empty string, or an empty array skips the phase and records `{ ran: 0, ok: 0, dropped: 0 }` with empty results.
45 +- any other value runs the phase.
46 +- a skipped phase remains addressable through empty `{PHASE.results}` and `{PHASE.failures}` arrays.
47 +- there are no loops or arbitrary expressions.
50 -`over` is either a literal array or a single interpolation token string (e.g.
51 -`"{review.results[].findings[]}"`).
49 +- `over` is a literal array or one interpolation token string, such as `"{review.results[].findings[]}"`.
50 +- `return` and `report` use the same single-selector grammar and resolve after all phases.
53 -### Fan-out mutation ownership
52 +### fan-out mutation ownership
55 -Without `writeIsolation`, a `fanout` step's initial tools must stay within Pi's core set:
56 -`read`, `grep`, `find`, `ls`, `bash`.
57 -The prompt must instruct the agent not to mutate anything.
58 -With `writeIsolation`, mutating tools are allowed and the non-empty rationale plus prompt must assign disjoint ownership to every item.
59 -Ultra trusts that declaration; it does not constrain paths or arbitrary custom mutators.
60 -Loaded extensions remain trusted to activate additional tools.
61 -Every listed tool must be available to sub-agents or Ultra refuses to start the workflow.
54 +- without `writeIsolation`, a `fanout` step's initial tools must stay within Pi's core set: `read`, `grep`, `find`, `ls`, and `bash`.
55 +- the prompt must instruct the agent not to mutate anything.
56 +- with `writeIsolation`, mutating tools are allowed; the non-empty rationale and prompt must assign disjoint ownership to every item.
57 +- Ultra trusts that declaration and does not constrain paths or arbitrary custom mutators.
58 +- loaded extensions remain trusted to activate additional tools.
59 +- every listed tool must be available to sub-agents or Ultra refuses to start the workflow.
63 -## Steps
61 +## step semantics
65 -| Field | Required | Type | Meaning |
66 -| --------------- | -------- | --------------------- | ------------------------------------------------------------------------------------------------- |
67 -| `summary` | yes | string | One short progress-UI sentence describing the task; interpolates `{...}` tokens. |
68 -| `prompt` | yes | string | The exact agent instruction; interpolates `{...}` tokens (see below). |
69 -| `tools` | no | string[] | Initial tool set. Omit when the agent needs only the injected result-submission tool. |
70 -| `model` | no | string | Configured tier name or concrete `provider/model`; omitted uses the session model. |
71 -| `thinkingLevel` | no | off…max | Optional explicit reasoning override; normally omit it for the tier default. |
72 -| `dynamicExtensions` | no | string[] \| selector | Stable catalog names, or one selector resolving to names, loaded in declared order. |
73 -| `schema` | no | string \| JSON Schema | A `schemas` name, or an inline schema that validates structured output. |
63 +- schema-declared step fields and types are available through the active tool contract.
64 +- `summary` is rendered on tool and overlay agent rows.
65 +- `prompt` is the exact resolved child instruction.
66 +- model, tools, thinking-level, dynamic-extension, and output-schema behavior below describes runtime semantics not captured by field shape alone.
67 +- use the same interpolation grammar as `prompt` so fan-out rows can identify their item.
68 +- summary interpolation renders objects as short labels and arrays as counts; prompts retain compact JSON.
69 +- the overlay shows the exact resolved `prompt` as the first user message in that agent's transcript.
70 +- normal child context-file loading is enabled; children do not inherit the parent conversation.
71 +- prompts must supply the task context, constraints, instruction paths, evidence, and authority boundaries each child needs.
72 +- see [child context mechanics](../ultra-authoring/SKILL.md#give-children-context).
75 -`summary` is rendered on tool and overlay agent rows.
76 -Keep it concise and concrete; use the same interpolation grammar as `prompt` so fan-out rows can identify their item.
77 -Summary interpolation renders objects as short labels and arrays as counts; prompts retain compact JSON.
78 -The overlay separately shows the exact resolved `prompt` as the first user message in that agent's transcript.
79 -Normal child context-file loading is enabled; children do not inherit the parent conversation.
80 -Authored prompts must still supply task-specific parent constraints, applicable nested instruction paths, required evidence, and authority/blocker boundaries as required by the [delegation contract](../ultra-authoring/SKILL.md#delegation-contract).
81 -Provide tools to read those instructions and evidence; skill and reference contents remain on demand.
74 +- every step returns one JSON object through the terminating `structured_output` tool.
75 +- Ultra requests provider-side strict schema sampling when supported and always validates the submitted object locally.
76 +- when an agent stops without submitting, Ultra sends up to `ultra.maxRetries` focused reminders in the same session.
77 +- without `schema`, any JSON object is accepted.
78 +- `schema` validates that object and is required when downstream phases depend on a stable shape.
79 +- a string `schema` is a name reference and must resolve in the spec's `schemas` map.
83 -Every step returns a JSON object through the terminating `structured_output` tool.
84 -Ultra requests provider-side strict schema sampling when supported and always validates the submitted object locally.
85 -When an agent stops without submitting, Ultra sends up to `ultra.maxRetries` focused reminders in the same session.
86 -Without `schema`, any JSON object is accepted.
87 -A `schema` validates that object and is required when downstream phases depend on a stable shape.
88 -A string `schema` is a **name reference** and must resolve in the spec's `schemas` map, or validation fails.
81 +- configured model tiers live in Pi settings under `ultra.modelTiers`.
82 +- tier names are resolved dynamically, so a step may select any configured tier name.
83 +- an unknown bare tier name is invalid; a concrete `provider/model` reference is always valid.
84 +- omitting `model`, or selecting a tier whose `model` is `null`, uses the session model.
85 +- an explicit `thinkingLevel` overrides the tier default.
90 -Configured model tiers live in Pi settings under `ultra.modelTiers`.
91 -Their names are resolved dynamically, so a step may select any configured tier name.
92 -An unknown bare tier name is invalid; a concrete `provider/model` reference is always valid.
93 -Omitting `model`, or selecting a tier whose `model` is `null`, uses the session model.
94 -An explicit `thinkingLevel` overrides the tier default, but workflow authors normally omit it.
87 +### dynamic extensions and builders
96 -### Dynamic extensions and builders
89 +- `dynamicExtensions` names stable catalog identities, never hashes.
90 +- Ultra resolves each name to one validated hash and a run-local source snapshot before child creation.
91 +- normal configured extensions load first, then dynamic extensions in declared order.
92 +- a builder step selects all four tools or none: `search_dynamic_extensions`, `create_dynamic_extension`, `checkout_dynamic_extension`, and `validate_dynamic_extension`.
93 +- builders receive every available child tool except `run_workflow` plus the [builder contract](../dynamic-extension-builder.md).
94 +- known canonical names may load directly; discovery, creation, improvement, stale validation, and repair require a builder.
95 +- a builder failure blocks every later phase with typed `dynamic_extension_builder_failure` outcomes.
98 -`dynamicExtensions` names stable catalog identities, never hashes.
99 -Immediately before child creation, Ultra resolves each name to one validated hash and a run-local source snapshot; name and hash participate in the checkpoint key.
100 -Normal configured extensions load first, then dynamic extensions in declared order.
101 -`subagentExtensions: "none"` suppresses normal extensions only.
102 -Generated tools activate during `session_start`.
103 -Dynamic extensions may declare exact qualified normal-registration overrides, but Ultra's workflow and builder tool names remain reserved.
97 +### model mechanics
105 -A builder step selects all four tools or none: `search_dynamic_extensions`, `create_dynamic_extension`, `checkout_dynamic_extension`, and `validate_dynamic_extension`.
106 -Builders receive every available child tool except `run_workflow` plus a versioned authoring contract.
107 -Available host capability is not authorized scope; the brief must preserve applicable constraints, instruction paths, required evidence, and authority/blocker boundaries.
108 -These authoring obligations do not change validator contract version 2 or its validation gates.
109 -A builder failure blocks every later phase with typed `u008_builder_failure_blocker` outcomes; no normal-tool fallback exists.
110 -Known canonical names may load directly, but discovery, creation, improvement, stale validation, and repair require a builder.
99 +- when `run_workflow` is active, Ultra injects the effective tier rubric generated from merged `ultra.modelTiers` settings into the orchestrator system prompt.
100 +- a step's `model` names a configured tier or a concrete `provider/model` reference.
101 +- an unknown bare tier name is invalid; omitting `model` uses the session model.
102 +- a tier supplies a default `thinkingLevel`; an explicit step value overrides it.
103 +- tier instructions guide the orchestrator and are not passed to child agents.
112 -### Model tier and thinking policy
105 +## interpolation grammar
114 -When `run_workflow` is active, Ultra injects one effective tier rubric generated from merged `ultra.modelTiers` settings into the orchestrator system prompt.
115 -Use that rubric to choose each step's explicit tier selection; do not encode a hardcoded tier-to-role list here or in child prompts.
116 -Choose the least-compute tier likely to satisfy each step's acceptance criteria.
117 -Escalate only for a concrete remaining reasoning bottleneck that a lower tier cannot adequately handle.
118 -Broad scope, long prompts, routine review or debugging, and general importance do not justify more compute.
119 -Supply missing context or report blocked instead of compensating with a larger tier.
120 -Saved workflows retain their selected tier names.
122 -Normally omit `thinkingLevel` so settings decide the tier default.
123 -Set it only when that step has a distinct, explicitly justified reasoning need, because the explicit value wins over the tier default.
124 -Tier instructions guide the orchestrator's selection and are never model-specific child instructions.
125 -If every phase selects the same tier, re-evaluate whether their work actually warrants it.
127 -## Interpolation grammar
129 -A closed mini-syntax — not an expression evaluator, no `eval`. Tokens
130 -(braces are literal here, shown without JSON quoting):
107 +- interpolation is a closed mini-syntax, not an expression evaluator, and has no `eval`.
108 +- braces below are literal and shown without JSON quoting:
132 110 ```text
133 111 {args.NAME} workflow input value
134 112 {item} current fanout item (string verbatim; else compact JSON)
135 113 {PHASE.results} a prior phase's results array (nulls retained positionally)
136 114 {PHASE.failures} a prior phase's typed failure array
137 -{PHASE.results[].FIELD[]} flatten: concat each result's FIELD array (nulls/missing skipped)
138 -{EXPR | where FIELD} keep elements whose boolean FIELD is truthy
115 +{PHASE.results[].FIELD[]} flatten: concat each result's FIELD array (nulls/missing skipped)
116 +{EXPR | where FIELD} keep elements whose boolean FIELD is truthy
117 +{EXPR | where FIELD == "VALUE"} keep elements whose string FIELD exactly equals VALUE
118 +{EXPR | groupBy FIELD} group records by exact string FIELD value
141 -Anything outside this grammar raises a validation error at run time. In a
142 -step `prompt`, the only literal braces allowed are these tokens — do not put
143 -raw `{`/`}` (e.g. an inline JSON example) in a prompt; describe the shape in
144 -words and use a `schema` instead.
121 +- `where FIELD` keeps records whose field is truthy.
122 +- `where FIELD == "VALUE"` keeps records whose field is a string exactly equal to VALUE; missing, null, and non-string fields do not match.
123 +- `groupBy FIELD` returns first-seen groups shaped as `{ "key": "...", "items": [...] }`; records with missing, null, or non-string fields are skipped.
124 +- group item order and group order follow the source array.
125 +- the left side of `where` and `groupBy` must resolve to an array.
126 +- anything outside this grammar raises a validation error at run time.
127 +- in a step `prompt`, literal braces are allowed only as these tokens.
128 +- describe inline JSON shapes in words and use a `schema` instead.
146 -### Explicit linear recovery
130 +### explicit linear recovery
148 -A dropped step appears in `{PHASE.failures}` with `agentId`, `index`, optional original `item`, `code`, `message`, `retryable`, `attempts`, optional `model`, optional `thinkingLevel`, optional accumulated `usage`, and an optional 2,000-character `transcriptTail`.
149 -Failure codes are `auth`, `transport`, `schema`, `missing-output`, `aborted`, or `session`.
151 -Use the non-empty failure array to condition and fan out one later recovery phase.
152 -Recovery must restate the original goal, scope, constraints, instruction paths, required evidence, tools, and output schema; failure diagnostics supplement the assignment, never replace it.
153 -This complete example reads only the root `package.json` to extract its declared package name and license, with one explicit recovery and a final selection that preserves blocked work.
154 -Supply `instructions` as a run input containing applicable parent constraints and instruction paths, or explicitly state that no additional task-specific instructions apply.
155 -No child may infer a license from repository conventions or change files to make the task succeed.
132 +- a dropped step appears in `{PHASE.failures}` with `agentId`, `index`, optional original `item`, `code`, `message`, `retryable`, `attempts`, optional `model`, optional `thinkingLevel`, optional accumulated `usage`, and an optional 2,000-character `transcriptTail`.
133 +- failure codes are `auth`, `transport`, `schema`, `missing-output`, `aborted`, `session`, or `dynamic_extension_builder_failure`.
134 +- use a non-empty failure array to condition and fan out one later recovery phase.
135 +- recovery must restate the original goal, scope, constraints, instruction paths, required evidence, tools, and output schema; failure diagnostics supplement the assignment, never replace it.
136 +- the example below reads only the root `package.json`, then recovers and selects an outcome that preserves blocked work.
137 +- supply `instructions` as a run input containing applicable parent constraints and instruction paths, or explicitly state that no additional task-specific instructions apply.
138 +- no child may infer a license from repository conventions or change files to make the task succeed.
157 140 ```json
216 -Ultra has no timeout, automatic rerun, loop, or back-edge.
217 -`ultra.maxRetries` sends result-submission reminders only after an agent turn ends without output; it never reruns the work in a fresh session.
218 -Workflow authors explicitly place any recovery phase after the failed phase.
199 +- Ultra has no timeout, automatic rerun, loop, or back-edge.
200 +- `ultra.maxRetries` sends result-submission reminders only after an agent turn ends without output; it never reruns the work in a fresh session.
201 +- workflow authors explicitly place any recovery phase after the failed phase.
220 -### Schema join-keys
203 +### schema join-keys
222 -When one phase fans out over another's findings, require a stable unique `id` in both schemas and instruct the consumer to echo it verbatim.
223 -Use the same ID-based join as the bundled review workflow: a review dimension, a colon, and a locally unique identifier containing only letters, digits, underscores, or hyphens, for example `correctness:1`.
224 -Keep IDs stable through verification and distinguish multiple findings in one file.
225 -`file` is a locator, not a unique join key; never merge or overwrite candidates merely because they share a file or title.
226 -Missing or duplicate verdict IDs remain unresolved work, not implicit refutations.
205 +- when one phase fans out over another's findings, require a stable unique `id` in both schemas and instruct the consumer to echo it verbatim.
206 +- use the same ID-based join as the bundled review workflow: a review dimension, a colon, and a locally unique identifier containing only letters, digits, underscores, or hyphens, such as `correctness:1`.
207 +- keep IDs stable through verification and distinguish multiple findings in one file.
208 +- `file` is a locator, not a unique join key; never merge or overwrite candidates merely because they share a file or title.
209 +- missing or duplicate verdict IDs remain unresolved work, not implicit refutations.
228 -## Worked example — minimal single fan-out
211 +## example gallery
230 -Every `fanout` agent reads one path and returns a structured summary. Tools are
231 -read-only; the prompt forbids mutation.
213 +The gallery intentionally exercises the structural variants and selector forms accepted by the runtime.
214 +Each block is independently parsed and validated by the workflow reference tests.
216 +### tiny single
218 +```json
220 + "name": "tiny-single",
221 + "phases": [
223 + "id": "answer",
224 + "kind": "single",
225 + "step": { "summary": "Answer the request", "prompt": "Answer the request." }
229 +```
231 +### single with positional input
233 +```json
235 + "name": "single-input",
236 + "args": { "hint": "<question>", "params": ["question"] },
237 + "phases": [
239 + "id": "answer",
240 + "kind": "single",
241 + "step": { "summary": "Answer the question", "prompt": "Answer: {args.question}" }
243 + ],
244 + "return": "{answer.results}"
246 +```
248 +### literal fanout
250 +```json
252 + "name": "literal-fanout",
253 + "phases": [
255 + "id": "inspect",
256 + "kind": "fanout",
257 + "over": ["README.md", "CHANGELOG.md", "package.json"],
258 + "step": { "summary": "Inspect {item}", "prompt": "Read {item} and summarize it." }
260 + ],
261 + "return": "{inspect.results}"
263 +```
265 +### input-driven fanout
267 +```json
269 + "name": "input-fanout",
270 + "args": { "hint": "<paths>", "params": ["paths"] },
271 + "phases": [
273 + "id": "inspect",
274 + "kind": "fanout",
275 + "over": "{args.paths}",
276 + "step": { "summary": "Inspect {item}", "prompt": "Read {item}; return its key facts." }
280 +```
282 +### ignored single over
284 +```json
286 + "name": "single-over",
287 + "phases": [
289 + "id": "one",
290 + "kind": "single",
291 + "over": ["ignored", "also-ignored"],
292 + "step": { "summary": "Run once", "prompt": "Run this synthesis once." }
296 +```
298 +### conditional single
300 +```json
302 + "name": "conditional-single",
303 + "args": { "hint": "optional input", "params": ["input"] },
304 + "phases": [
306 + "id": "collect",
307 + "kind": "single",
308 + "step": { "summary": "Collect evidence", "prompt": "Collect evidence for {args.input}." }
309 + },
311 + "id": "judge",
312 + "kind": "single",
313 + "when": "{collect.results}",
314 + "step": { "summary": "Judge evidence", "prompt": "Judge {collect.results}." }
316 + ],
317 + "return": "{judge.results}"
319 +```
321 +### conditional fanout
323 +```json
325 + "name": "conditional-fanout",
326 + "phases": [
328 + "id": "discover",
329 + "kind": "single",
330 + "step": { "summary": "Discover targets", "prompt": "Discover targets." }
331 + },
333 + "id": "verify",
334 + "kind": "fanout",
335 + "when": "{discover.results}",
336 + "over": "{discover.results}",
337 + "step": { "summary": "Verify {item}", "prompt": "Verify target {item}." }
339 + ],
340 + "return": "{verify.results}"
342 +```
344 +### flatten selector
346 +```json
348 + "name": "flatten-findings",
349 + "phases": [
351 + "id": "review",
352 + "kind": "fanout",
353 + "over": ["correctness", "security"],
354 + "step": { "summary": "Review {item}", "prompt": "Review the {item} dimension." }
355 + },
357 + "id": "verify",
358 + "kind": "fanout",
359 + "over": "{review.results[].findings[]}",
360 + "step": { "summary": "Verify {item}", "prompt": "Verify finding {item}." }
362 + ],
363 + "return": "{verify.results}"
365 +```
367 +### where selector
369 +```json
371 + "name": "truthy-filter",
372 + "phases": [
374 + "id": "candidates",
375 + "kind": "fanout",
376 + "over": ["a", "b", "c"],
377 + "step": { "summary": "Inspect {item}", "prompt": "Inspect candidate {item}." }
378 + },
380 + "id": "selected",
381 + "kind": "single",
382 + "step": { "summary": "Select candidates", "prompt": "Select from {candidates.results | where valid}." }
384 + ],
385 + "return": "{selected.results}"
387 +```
389 +### exact string filter
391 +```json
393 + "name": "source-filter",
394 + "phases": [
396 + "id": "evidence",
397 + "kind": "fanout",
398 + "over": ["sessions", "repository", "web"],
399 + "step": { "summary": "Collect {item} evidence", "prompt": "Collect evidence from {item}." }
400 + },
402 + "id": "sessions",
403 + "kind": "fanout",
404 + "when": "{evidence.results | where source_type == \"sessions\"}",
405 + "over": "{evidence.results | where source_type == \"sessions\"}",
406 + "step": { "summary": "Inspect session evidence {item}", "prompt": "Inspect session evidence {item}." }
408 + ],
409 + "return": "{sessions.results}"
411 +```
413 +### dynamic grouping
415 +```json
417 + "name": "source-groups",
418 + "phases": [
420 + "id": "evidence",
421 + "kind": "fanout",
422 + "over": ["one", "two", "three"],
423 + "step": { "summary": "Collect evidence for {item}", "prompt": "Collect evidence for {item}." }
424 + },
426 + "id": "groups",
427 + "kind": "fanout",
428 + "over": "{evidence.results | groupBy source_type}",
429 + "step": { "summary": "Process source group {item}", "prompt": "Process this source group: {item}." }
431 + ],
432 + "return": "{groups.results}"
434 +```
436 +### named output schema
438 +```json
440 + "name": "named-schema",
441 + "schemas": {
442 + "Finding": {
443 + "type": "object",
444 + "properties": { "title": { "type": "string" } },
445 + "required": ["title"]
447 + },
448 + "phases": [
450 + "id": "find",
451 + "kind": "fanout",
452 + "over": ["src/a.ts", "src/b.ts"],
453 + "step": { "summary": "Find issues in {item}", "prompt": "Inspect {item}.", "schema": "Finding" }
457 +```
459 +### inline output schema
461 +```json
463 + "name": "inline-schema",
464 + "phases": [
466 + "id": "measure",
467 + "kind": "single",
468 + "step": {
469 + "summary": "Measure the target",
470 + "prompt": "Measure the target and return its size.",
471 + "schema": {
472 + "type": "object",
473 + "properties": { "size": { "type": "number", "minimum": 0 } },
474 + "required": ["size"]
480 +```
482 +### explicit concurrency
484 +```json
486 + "name": "bounded-fanout",
487 + "phases": [
489 + "id": "inspect",
490 + "kind": "fanout",
491 + "over": [1, 2, 3, 4, 5, 6],
492 + "concurrency": 2,
493 + "step": { "summary": "Inspect item {item}", "prompt": "Inspect item {item}." }
495 + ],
496 + "return": "{inspect.results}"
498 +```
500 +### explicit model and reasoning
502 +```json
504 + "name": "routed-review",
505 + "phases": [
507 + "id": "review",
508 + "kind": "single",
509 + "step": {
510 + "summary": "Review the change",
511 + "prompt": "Review the supplied change carefully.",
512 + "model": "large",
513 + "thinkingLevel": "high"
518 +```
520 +### concrete provider model
522 +```json
524 + "name": "concrete-model",
525 + "phases": [
527 + "id": "extract",
528 + "kind": "single",
529 + "step": {
530 + "summary": "Extract the value",
531 + "prompt": "Extract the requested value.",
532 + "model": "provider/model-id",
533 + "thinkingLevel": "low"
536 + ],
537 + "return": "{extract.results}"
539 +```
541 +### read-only fanout tools
543 +```json
545 + "name": "read-only-review",
546 + "phases": [
548 + "id": "review",
549 + "kind": "fanout",
550 + "over": ["correctness", "security"],
551 + "tools": ["read", "grep", "find", "ls", "bash"],
552 + "step": { "summary": "Review {item}", "prompt": "Review {item}; do not mutate anything." }
556 +```
558 +### isolated mutation fanout
560 +```json
562 + "name": "isolated-edits",
563 + "phases": [
565 + "id": "edit",
566 + "kind": "fanout",
567 + "over": ["src/a.ts", "src/b.ts"],
568 + "writeIsolation": "Each item owns exactly one disjoint source file.",
569 + "tools": ["read", "edit"],
570 + "step": { "summary": "Update {item}", "prompt": "Update only {item}; it is exclusively yours." }
574 +```
576 +### dynamic extension consumer
578 +```json
580 + "name": "dynamic-consumer",
581 + "args": { "hint": "<extension names>", "params": ["extensions"] },
582 + "phases": [
584 + "id": "inspect",
585 + "kind": "single",
586 + "step": {
587 + "summary": "Inspect with extensions",
588 + "prompt": "Inspect the task with the requested capabilities.",
589 + "dynamicExtensions": "{args.extensions}"
594 +```
596 +### dynamic extension builder
598 +```json
600 + "name": "dynamic-builder",
601 + "phases": [
603 + "id": "build",
604 + "kind": "single",
605 + "step": {
606 + "summary": "Build the requested capability",
607 + "prompt": "Discover or build the requested capability using the builder contract.",
608 + "tools": ["search_dynamic_extensions", "create_dynamic_extension", "checkout_dynamic_extension", "validate_dynamic_extension"]
610 + },
612 + "id": "use",
613 + "kind": "single",
614 + "step": { "summary": "Use the capability", "prompt": "Use the validated capability." }
616 + ],
617 + "return": "{use.results}"
619 +```
621 +### independent human report
623 +```json
625 + "name": "separate-report",
626 + "phases": [
628 + "id": "work",
629 + "kind": "single",
630 + "step": { "summary": "Do the full work", "prompt": "Produce complete machine-readable evidence." }
631 + },
633 + "id": "human",
634 + "kind": "single",
635 + "step": { "summary": "Prepare the report", "prompt": "Prepare a concise human report from {work.results}." }
637 + ],
638 + "return": "{work.results}",
639 + "report": "{human.results}"
641 +```
643 +### explicit report selector
645 +```json
647 + "name": "report-projection",
648 + "phases": [
650 + "id": "work",
651 + "kind": "single",
652 + "step": { "summary": "Produce report data", "prompt": "Produce report data." }
654 + ],
655 + "return": "{work.results}",
656 + "report": "{work.results}"
658 +```
660 +### empty fanout
662 +```json
664 + "name": "empty-fanout",
665 + "phases": [
667 + "id": "nothing",
668 + "kind": "fanout",
669 + "over": [],
670 + "step": { "summary": "Process {item}", "prompt": "Process {item}." }
672 + ],
673 + "return": "{nothing.results}"
675 +```
677 +### recovery phase
679 +```json
681 + "name": "recover-failure",
682 + "phases": [
684 + "id": "first",
685 + "kind": "fanout",
686 + "over": ["one", "two"],
687 + "step": { "summary": "Try {item}", "prompt": "Try the task for {item}." }
688 + },
690 + "id": "recover",
691 + "kind": "fanout",
692 + "when": "{first.failures}",
693 + "over": "{first.failures}",
694 + "step": { "summary": "Recover {item}", "prompt": "Recover the failed task described by {item}." }
696 + ],
697 + "return": "{recover.results}",
698 + "report": "{first.failures}"
700 +```
702 +### arbitrary input metadata
704 +```json
706 + "name": "metadata-input",
707 + "args": { "hint": "<task>", "params": ["task"], "owner": "workflow-team", "labels": ["safe", "read-only"] },
708 + "phases": [
710 + "id": "task",
711 + "kind": "single",
712 + "step": { "summary": "Complete the task", "prompt": "Complete {args.task}." }
716 +```
718 +### three-stage pipeline
720 +```json
722 + "name": "three-stage",
723 + "args": { "hint": "<target>", "params": ["target"] },
724 + "phases": [
726 + "id": "research",
727 + "kind": "single",
728 + "step": { "summary": "Research {args.target}", "prompt": "Research {args.target}." }
729 + },
731 + "id": "plan",
732 + "kind": "single",
733 + "step": { "summary": "Plan the work", "prompt": "Create a plan from {research.results}." }
734 + },
736 + "id": "check",
737 + "kind": "single",
738 + "step": { "summary": "Check the plan", "prompt": "Check the plan in {plan.results}." }
740 + ],
741 + "return": "{check.results}",
742 + "report": "{plan.results}"
744 +```
746 +### multi-fanout pipeline
748 +```json
750 + "name": "multi-fanout",
751 + "phases": [
753 + "id": "files",
754 + "kind": "fanout",
755 + "over": ["a.ts", "b.ts", "c.ts"],
756 + "step": { "summary": "Inspect {item}", "prompt": "Inspect {item}." }
757 + },
759 + "id": "tests",
760 + "kind": "fanout",
761 + "over": "{files.results}",
762 + "step": { "summary": "Test {item}", "prompt": "Test the evidence for {item}." }
763 + },
765 + "id": "summary",
766 + "kind": "single",
767 + "step": { "summary": "Summarize the pipeline", "prompt": "Summarize {files.results} and {tests.results}." }
769 + ],
770 + "return": "{summary.results}"
772 +```
774 +### complex mixed workflow
776 +```json
778 + "name": "complex-mixed",
779 + "description": "Research, filter, verify, recover, and report a multi-dimensional review.",
780 + "args": { "hint": "<target>", "params": ["target", "instructions"] },
781 + "schemas": {
782 + "Candidate": {
783 + "type": "object",
784 + "properties": { "id": { "type": "string" }, "valid": { "type": "boolean" } },
785 + "required": ["id", "valid"]
786 + },
787 + "Verdict": {
788 + "type": "object",
789 + "properties": { "id": { "type": "string" }, "status": { "type": "string" } },
790 + "required": ["id", "status"]
792 + },
793 + "phases": [
795 + "id": "research",
796 + "kind": "fanout",
797 + "over": ["correctness", "security", "performance"],
798 + "concurrency": 2,
799 + "tools": ["read", "grep", "find", "ls", "bash"],
800 + "step": { "summary": "Research {item}", "prompt": "Research {item} for {args.target}; {args.instructions}.", "schema": "Candidate", "model": "large" }
801 + },
803 + "id": "verify",
804 + "kind": "fanout",
805 + "when": "{research.results}",
806 + "over": "{research.results | where valid}",
807 + "step": { "summary": "Verify {item}", "prompt": "Verify candidate {item}.", "schema": "Verdict", "model": "large", "thinkingLevel": "high" }
808 + },
810 + "id": "recover",
811 + "kind": "fanout",
812 + "when": "{verify.failures}",
813 + "over": "{verify.failures}",
814 + "step": { "summary": "Recover {item}", "prompt": "Recover failed verification {item}.", "schema": "Verdict", "model": "medium" }
815 + },
817 + "id": "report",
818 + "kind": "single",
819 + "step": { "summary": "Write the final report", "prompt": "Write a report from {research.results}, {verify.results}, and {recover.results}." }
821 + ],
822 + "return": "{verify.results}",
823 + "report": "{report.results}"
825 +```
827 +## worked example — minimal single fan-out
829 +- every `fanout` agent reads one path and returns a structured summary.
830 +- tools are read-only and the prompt forbids mutation.
233 832 ```json
263 -## Worked example: two phases with a unique finding join key
862 +## worked example: two phases with a unique finding join key
265 -Fan out a review across dimensions, then fan out verification over the flattened findings.
266 -`Findings.findings[].id` → `Verdict.id` is the join key, even for several findings in one file.
267 -The single `params` entry maps `/ultra run review main` to `{args.base}`; an empty base selects `git diff`, the unstaged working-tree diff.
268 -A base ref selects the current tracked working tree against that ref, not an unrelated comparison chosen by the verifier.
269 -Supply applicable task-specific parent constraints and instruction paths through the `instructions` input when constructing or invoking this example with JSON arguments.
270 -Verification receives that same original base, constraints, candidate detail, and evidence, plus explicit inspection tools.
271 -`bash` is authorized only for read-only Git inspection, not arbitrary commands or tests that write caches.
272 -The selected output preserves every verdict, including unresolved ones; the closed selector grammar cannot compare status strings and must not filter uncertainty away.
273 -Typed phase failures accompany the selected output, so missing verifier results remain visible as incomplete work.
864 +- fan out a review across dimensions, then fan out verification over the flattened findings.
865 +- `Findings.findings[].id` → `Verdict.id` is the join key, including several findings in one file.
866 +- the single `params` entry maps `/ultra run review main` to `{args.base}`; an empty base selects the unstaged working-tree diff.
867 +- a base ref selects the current tracked working tree against that ref, not an unrelated comparison chosen by the verifier.
868 +- supply task-specific parent constraints and instruction paths through `instructions` when constructing or invoking this example with JSON arguments.
869 +- verification receives the original base, constraints, candidate detail, and evidence, plus explicit inspection tools.
870 +- `bash` is authorized only for read-only Git inspection, not arbitrary commands or tests that write caches.
871 +- the selected output preserves every verdict, including unresolved ones; the closed selector grammar cannot compare status strings or filter uncertainty away.
872 +- typed phase failures accompany the selected output, so missing verifier results remain visible as incomplete work.
275 874 ```json
349 -## Where workflows live
948 +## where workflows live
351 -Discovery merges three locations, lowest-to-highest precedence — a later source
352 -overrides a bundled name:
354 -1. **bundled** — `<extension>/workflows/*.json` (ships with ultra)
355 -2. **global** — `~/.pi/agent/workflows/*.json`
356 -3. **project** — `<cwd>/.pi/workflows/*.json`
358 -Save an authored workflow as `<name>.json` (filename should match the spec's
359 -`name`) in the project dir for repo-scoped workflows, or the global dir for
360 -personal ones.
950 +- discovery merges three locations, lowest-to-highest precedence; a later source overrides a bundled name.
951 +- **bundled** — `<extension>/workflows/*.json` (ships with Ultra)
952 +- **global** — `~/.pi/agent/workflows/*.json`
953 +- **project** — `<cwd>/.pi/workflows/*.json`
954 +- save an authored workflow as `<name>.json`, with a filename matching the spec's `name`, in the project directory for repo-scoped workflows or the global directory for personal workflows.

extensions/ultra/prompts/ultra-authoring/SKILL.md

Modified
2 2 name: ultra-authoring
3 -description: "Use before authoring or editing Ultra workflows: phases, handoffs, schemas, interpolation, model selection, and dynamic extensions."
3 +description: "use before authoring or editing Ultra workflow specs or calling run_workflow."
6 -# Authoring ultra workflows
6 +# authoring Ultra workflows
8 -These instructions help you choose between running an **ephemeral one-off**
9 -ultra workflow and co-designing a **persisted** workflow the user can re-run as
10 -`/ultra run <name>`.
11 -This skill and the [canonical spec reference](../references/spec-format.md) are available on demand, not injected by `/ultra exec`.
12 -Only the active `run_workflow` tool advertises this skill's path; it is not globally discovered or registered as a `/skill` command.
13 -The tool accepts `spec` as a generic object; its workflow structure is deliberately omitted from the always-visible tool schema.
14 -Before constructing or editing a spec, read the canonical reference once for required fields, selectors, constraints, and examples.
15 -Reuse the loaded guidance for subsequent calls; runtime validation still checks the complete workflow before agents run.
8 +read the [workflow reference](../references/spec-format.md) before constructing
9 +or editing a workflow spec.
10 +read the [dynamic-extension builder contract](../dynamic-extension-builder.md)
11 +when a workflow discovers, creates, improves, validates, or repairs an extension.
17 -## First: persist vs. one-off
13 +## choose the command
19 -Infer the user's intent from the request and its context.
20 -Choose a **persisted** workflow when durable reuse is explicit or strongly implied, such as repeated use, a named utility, sharing, or future `/ultra run` invocations.
21 -Choose an **ephemeral one-off** when the request seeks one bounded result now and gives no meaningful reuse signal.
22 -Do not treat `/ultra exec` itself or generic creation verbs as evidence for persistence.
15 +- `/ultra exec <instructions>` sends an inline request and does not save a workflow.
16 +- `/ultra run <name> [args]` runs a saved workflow.
17 +- `run_workflow` accepts a saved `name` or an inline `spec` object.
18 +- use an inline spec unless the user explicitly requests a reusable saved workflow.
19 +- save project workflows as `<cwd>/.pi/workflows/<name>.json`.
20 +- save personal workflows as `~/.pi/agent/workflows/<name>.json`.
21 +- ask for scope only when project versus personal ownership is unclear.
24 -When confidence is high, act on that choice without asking or announcing the routing.
25 -When confidence is not high, ask one focused question before invoking `run_workflow` or writing a workflow file.
26 -Never resolve uncertainty by defaulting to persistence or ephemerality.
23 +## build the spec
28 -## Single-agent gate
25 +- a spec is one JSON object with a `name` and at least one phase.
26 +- phases run in order and use `single` for one child or `fanout` for one child per `over` item.
27 +- `fanout` requires `over`; `single` ignores `over`.
28 +- phase IDs are unique identifiers; `args` is reserved.
29 +- `when` skips a phase when its selector resolves to a false, null, undefined, empty string, or empty array value.
30 +- `return` selects the final value; without it, the last phase's results are returned.
31 +- `report` optionally selects the human-facing value independently from `return`.
32 +- use `args` for declared run inputs and `schemas` for output shapes reused by later phases.
30 -Use ultra only when independent parallel work, staged agent handoffs, or isolated specialist contexts materially improve the result.
31 -Otherwise perform the task directly and skip ultra's orchestration overhead.
34 +## give children context
33 -## Delegation contract
36 +children run in separate sessions and do not inherit the parent conversation.
37 +supply each prompt with the task context, constraints, instruction paths, evidence,
38 +and authority boundaries it needs.
39 +normal child context-file loading remains enabled, but it does not replace that
40 +prompt-specific context.
35 -Optimize the total tokens and cost of producing an acceptable artifact within model context limits, not the size of each child's prompt.
36 -Never discard binding user instructions or decision-relevant evidence merely to save tokens.
37 -Children do not inherit the parent conversation; supply the goal, applicable constraints, settled decisions and their relevant rationale, necessary evidence, and completion criteria explicitly.
38 -Normal child context-file loading is enabled, but it does not inherit task-specific parent constraints or discover every nested instruction automatically.
39 -For repository work, supply applicable `AGENTS.md` paths, including nested instructions for the assigned scope, and require children to read those instructions before acting; provide the tools needed to access them and the task's evidence.
40 -Authored prompts must preserve applicable parent instructions, including prohibitions, required checks, and reporting obligations, not just a paraphrase of the goal.
41 -Define what each child may decide and what it must report as blocked rather than guess.
42 -Available host capability is not authorized scope; tools and loaded extensions never expand the assignment's authority.
42 +## connect phases
44 -Use the effective tier rubric injected into this orchestrator's system prompt to match capability to the decisions still required, not prompt length or phase position.
45 -Resolve ambiguity, choose the approach, and partition work before delegating execution to a tier intended for settled assignments.
46 -Selecting what matters is a judgment task, not merely text transformation; delegate mechanical transformations only after relevance and preservation rules are explicit.
47 -Before dispatch, ask: can this worker finish without guessing the user's intent or making decisions beyond its assigned capability?
48 -If not, add context, resolve the decision first, or select a more capable tier.
44 +use only the closed interpolation grammar in `summary`, `prompt`, `over`, `when`,
45 +`return`, and `report`.
46 +use `{item}` for the current fanout item and phase selectors for prior results or
47 +failures.
48 +keep literal braces out of child prompts unless they are interpolation tokens.
49 +`where FIELD` filters truthy fields; `where FIELD == "VALUE"` filters exact strings.
50 +`groupBy FIELD` returns first-seen `{key, items}` groups for generic fanout.
51 +use a `schema` when a later phase consumes a stable output shape.
52 +when joining findings to later results, carry a stable ID through both schemas.
50 -Use the fewest phases that deliver the outcome.
51 -Audit fanout work for overlap; merge redundant assignments unless independent checks are intentional.
52 -Prefer self-contained `{item}` payloads over repeating entire prior-phase results in every child; shared context is justified when each worker genuinely needs it.
53 -Pass the latest authoritative handoff and necessary evidence; keep bulk evidence in files with precise paths or locators rather than forcing workers to rediscover it.
54 -Shape producer schemas around downstream needs: concise decisions, stable join keys, and evidence references.
55 -Use task-sized `maxItems` and `maxLength` bounds; preserve blockers and uncertainty, and report overflow or gaps instead of silently discarding findings.
56 -Require verification to carry the original target, stable unique finding IDs, concrete evidence, and explicit confirmed/refuted/unresolved verdicts with reasons; uncertainty is not refutation.
57 -Gate writes on an explicit ready assignment, not the existence of a schema-valid plan.
58 -Final reports must preserve completed, blocked, and unresolved outcomes, including missing or skipped prerequisite work; never infer completion from an empty verification list.
59 -Judge completed artifacts against the user's goal and constraints; schema-valid output alone does not establish quality or completion.
54 +## control tools
61 -Select a concise final value with `return`; intermediate `phaseResults` stay out of model-facing envelopes.
62 -For both tool and command results, the full aggregate is always stored once and every bounded model-facing envelope includes `fullOutputPath`.
63 -Phase counts, bounded failure diagnostics, and usage accompany the selected value when it fits; oversized values use a retrieval notice within the 50KB or 2000 lines limit.
64 -Complete contents, including intermediate results and failure inputs/transcript tails, remain in UI details.
65 -Read that file selectively when needed; do not load the whole aggregate back into context.
66 -After `run_workflow` returns, synthesize its aggregate result into the user-facing response.
56 +all listed tools must be available to children or Ultra refuses to start.
57 +loaded extensions may activate additional tools.
58 +without `writeIsolation`, a fanout step uses only `read`, `grep`, `find`, `ls`, and
59 +`bash`, and its prompt forbids mutation.
60 +with `writeIsolation`, the prompt assigns disjoint ownership to every fanout item.
61 +every child returns one JSON object through `structured_output`.
68 -## Dynamic-extension delegation
63 +## use dynamic extensions
70 -Delegate one builder when a reusable extension can encode specialized capability, deterministic operations, or repeated reasoning that materially improves the user's task.
71 -A builder step selects all four tools: `search_dynamic_extensions`, `create_dynamic_extension`, `checkout_dynamic_extension`, and `validate_dynamic_extension`.
72 -Its brief must state topic, outcome, intended consumers, done conditions, applicable constraints, instruction paths, required evidence, and authority/blocker boundaries.
73 -These are obligations, not optional context; state explicitly when none apply rather than omitting them.
74 -Supply settled decisions and relevant rationale; include assumptions and platform facts when they affect the task.
75 -Require the builder to read applicable instructions and evidence before acting, stay within authorized scope despite available host capability, and report blocked when necessary context or authority is missing.
76 -One builder owns one extension.
77 -Independent builder topics may fan out only with a non-empty `writeIsolation` rationale assigning disjoint ownership.
78 -Use the effective tier rubric injected into the orchestrator system prompt to select a tier.
79 -Do not prescribe a model-specific tier or thinking level to the builder child.
80 -Known canonical names may be placed directly in a consumer's `dynamicExtensions`; discovery, creation, improvement, stale validation, and repair require a builder.
81 -Builder failure blocks later phases; never design a normal-tool fallback.
65 +`dynamicExtensions` names validated catalog extensions by stable identity.
66 +known names load into the child in declared order.
67 +discovery, creation, improvement, validation, and repair use one builder step with
68 +all four tools: `search_dynamic_extensions`, `create_dynamic_extension`,
69 +`checkout_dynamic_extension`, and `validate_dynamic_extension`.
70 +the builder receives the linked contract; a builder failure blocks later phases with code `dynamic_extension_builder_failure`.
83 -## Authoring flow (persisted)
72 +## select models
85 -1. **Understand the goal.** What does each stage do, and what flows between
86 - stages? Most workflows are _fan-out then synthesize_: parallel agents each
87 - handle one item (a dimension, a file, a finding), then a `single` pass
88 - judges or merges. Sketch the phases before writing JSON.
74 +`model` resolves a configured tier or a concrete `provider/model` reference.
75 +omitting `model` uses the session model.
76 +`thinkingLevel` overrides the selected tier's default.
77 +an unknown bare tier name fails validation.
90 -2. **Co-design the phases.** For each phase decide:
91 - - Give every phase a unique interpolation identifier; the reserved phase ID `args` is forbidden.
92 - - `single` (one synthesis/judge pass) vs. `fanout` (one agent per item).
93 - - `fanout` needs an `over` — a literal array, or an interpolation token over
94 - a prior phase's results (e.g. `{review.results[].findings[]}`).
95 - - Optional `when` uses one existing selector; empty/false skips the phase and produces empty results.
96 - - A recovery fan-out may use `{PHASE.failures}` for both `when` and `over`; recovery remains one explicit later phase, never a loop.
97 - Restate the original goal, scope, constraints, instruction paths, evidence, tools, and output schema; failure diagnostics supplement the assignment, never replace it.
98 - - The step's short `summary`, exact `prompt`, `tools`, and how tightly its structured JSON output must be constrained by a schema.
99 - - `summary` is one concise progress-UI sentence, uses the same interpolation grammar as `prompt`, and must describe the concrete sub-agent task without paraphrasing its result.
100 - Summary interpolation renders objects as short labels and arrays as counts; prompts retain compact JSON.
79 +## inspect results
102 -3. **Declare fan-out ownership.** A `fanout` step without `writeIsolation` stays
103 - within Pi's core `read`, `grep`, `find`, `ls`, and `bash` tools, and its prompt
104 - must forbid mutation because parallel writers race.
105 - A mutating fan-out requires a non-empty `writeIsolation` rationale and a prompt assigning disjoint ownership to each item.
106 - Loaded extensions remain trusted to activate additional tools.
107 - Every listed tool must be installed and available or Ultra refuses to start the workflow.
81 +`return` selects the value shown to the orchestrator.
82 +`report` selects the value shown to the human; unselected data remains agent-facing.
83 +the result includes failures, usage, and `fullOutputPath` for complete evidence.
84 +read that path selectively when the bounded result is insufficient.
85 +a dropped step appears in its phase's failures and can feed one later recovery phase;
86 +there are no automatic loops.
87 +execution status describes execution, not artifact quality or semantic success.
109 -4. **Design schemas and join-keys.** Give a step a `schema` (named in `schemas`,
110 - or inline) when a later phase consumes its output. When phase B fans out over
111 - phase A's findings, add a shared field (a join-key) to both schemas and tell
112 - B to echo it, so each result ties back to its source. See the `review`
113 - worked example in the reference.
89 +## save and run
115 -5. **Wire interpolation.** Use only the closed grammar in both `summary` and `prompt` (`{args.NAME}`, `{item}`,
116 - `{PHASE.results}`, `{PHASE.failures}`, `{PHASE.results[].FIELD[]}`, `{EXPR | where FIELD}`). Keep
117 - literal `{`/`}` out of prompts except these tokens. Set `return` to the
118 - selector for the workflow's final value.
120 -6. **Assign compute deliberately.** For every step:
121 - - Use the one effective tier rubric injected into this orchestrator's system prompt.
122 - It reflects the merged `ultra.modelTiers` settings, so do not copy a hardcoded tier-to-role list into a workflow.
123 - - Choose the least-compute tier likely to satisfy each step's acceptance criteria.
124 - Escalate only for a concrete remaining reasoning bottleneck that a lower tier cannot adequately handle.
125 - Broad scope, long prompts, routine review or debugging, and general importance do not justify more compute.
126 - Supply missing context or report blocked instead of compensating with a larger tier.
127 - - Save an explicit `model` tier selection when the step needs a tier; a concrete `provider/model` reference is also valid.
128 - An unknown bare tier name fails.
129 - Omit `model` when the session model is appropriate.
130 - - Saved workflows retain their explicit model selections; do not silently retier existing specs.
131 - - Normally omit `thinkingLevel` so the selected tier's settings default controls it.
132 - Set it only for a distinct, justified task need within the model's capabilities, because an explicit step value overrides the tier default.
133 - - Tier guidance is for this orchestrator's selection only.
134 - Do not put model-specific instructions in child prompts.
135 - - If every phase uses the same tier, challenge the assignment before saving.
137 -7. **Self-check the spec (there is no validator tool).** You can't call
138 - `parseWorkflow` directly — verify by inspection against the reference: every
139 - required field is present, every step has a short `summary`, every `fanout` has an `over`, every `when` is one selector, every string
140 - `schema` names a key in `schemas`, and interpolation uses only the closed
141 - grammar. The real validator is the load itself: saving and test-running
142 - (steps 8–9) parses the file through the runtime's `parseWorkflow`, which
143 - surfaces a TypeBox error naming the offending path if anything is malformed —
144 - fix and re-run until it loads clean. One caveat: a JSON _syntax_ error (a
145 - trailing comma, a missing quote) makes the file invisible to discovery
146 - entirely, so `/ultra run <name>` reports `unknown workflow` rather than a schema
147 - error — if a saved workflow seems to vanish, confirm it is valid JSON first.
149 -8. **Save it.** Write the spec as `<name>.json` (filename matching the spec's
150 - `name`):
151 - - project-scoped (lives in the repo): `<cwd>/.pi/workflows/<name>.json`
152 - - personal / global: `~/.pi/agent/workflows/<name>.json`
154 - Ask which scope only if it isn't clear from context (a repo-specific review
155 - workflow → project; a personal utility → global).
157 -9. **Offer a test run.** After saving, offer to run it: `/ultra run <name>` (with
158 - any `args` the spec's `hint` declares). Tighten prompts/schemas from what the
159 - first run surfaces.
161 -## Ephemeral one-off
163 -For a one-off, skip the persisted flow: construct the spec inline and call
164 -`run_workflow` with it as `spec`.
165 -Use the same format and rules (`spec-format.md`, fan-out ownership), but write
166 -nothing to disk.
167 -If the user later wants reuse, save it via the persisted flow.
91 +for a persisted workflow, write `<name>.json` in the selected workflow directory.
92 +the filename must match the spec's `name`.
93 +runtime validation checks the complete spec when it loads or runs.
94 +after saving, offer `/ultra run <name>` with any declared arguments.

extensions/ultra/result.ts

Modified
268 268 result: WorkflowResult,
269 269 expanded = false,
270 270 ): string {
271 + if (Object.hasOwn(result, "report"))
272 + return formatValueMarkdown(result.report);
271 273 const report = reviewReport(result);
272 274 const research = researchReport(result);
273 275 const selected = resultItems(result.result);

extensions/ultra/runner.ts

Modified
60 60 | "missing-output"
61 61 | "aborted"
62 62 | "session"
63 - | "u008_builder_failure_blocker";
63 + | "dynamic_extension_builder_failure";
65 65 export interface StepFailure {
66 66 code: FailureCode;
158 158 export interface SubagentSessionHandle {
159 159 manager: unknown;
160 - /** Stable child session identity used for U008 debug-log paths. */
160 + /** Stable child session identity used for dynamic-extension debug-log paths. */
161 161 id?: string;
162 162 /** True only when continuing a matching incomplete step transcript. */
163 163 resumed: boolean;
177 177 modelTiers?: ModelTiers;
178 178 /** Optional best-effort LLM summary for tool actions shown on the board. */
179 179 actionSummarizer?: ActionSummarizer;
180 - /** Lazy U008 builder tools, prompts, log paths, and snapshot metadata. */
180 + /** Lazy dynamic-extension builder tools, prompts, log paths, and snapshot metadata. */
181 181 dynamicExtensionResolver?: DynamicExtensionResolver;

extensions/ultra/settings.ts

Modified
31 31 tiny: {
32 32 model: null,
33 33 thinkingLevel: "low",
34 - instructions:
35 - "Labels, extraction, and formatting from supplied context. No open-ended investigation.",
34 + instructions: "No unresolved judgement.",
37 36 small: {
38 37 model: null,
39 38 thinkingLevel: "low",
40 - instructions:
41 - "Bounded scouting and straightforward edits with clear success criteria.",
39 + instructions: "Few local decisions; low consequence.",
43 41 medium: {
44 42 model: null,
45 43 thinkingLevel: "medium",
46 44 instructions:
47 - "Routine implementation and testing with prepared context and an agreed specification.",
45 + "Several related decisions; moderate ambiguity or consequence.",
49 47 large: {
50 48 model: null,
51 49 thinkingLevel: "high",
52 50 instructions:
53 - "Architecture, ambiguous debugging, critique, and synthesis requiring substantial judgment. Delegate independent work when it materially helps; keep verification scoped to changed behavior and required gates.",
51 + "Many dependent decisions; substantial ambiguity or consequence.",
55 53 huge: {
56 54 model: null,
57 55 thinkingLevel: "max",
58 56 instructions:
59 - "Reserved for exceptional reasoning bottlenecks that large cannot adequately handle; broad scope, routine review or debugging, and general importance are insufficient.",
57 + "Exceptional residual burden after decomposition, context, and verification.",
151 149 /** The sole tier rubric, rendered from the effective settings for each agent turn. */
152 150 export function modelTierPrompt(tiers: ModelTiers): string {
153 151 return [
154 - "# Ultra model tier policy",
155 - "Tier: model; default thinkingLevel; orchestrator-only selection guidance.",
156 - "Choose the least-compute tier likely to satisfy each step's acceptance criteria.",
157 - "Escalate only for a concrete remaining reasoning bottleneck; broad scope, long prompts, routine review or debugging, and general importance do not justify more compute.",
158 - "Supply missing context or report blocked instead of compensating with a larger tier.",
152 + "# ultra model tier policy",
153 + "tier: model; default thinkingLevel; orchestrator-only selection guidance.",
154 + "select by residual judgement; default to small; use tiny only when context determines the result.",
155 + "count unresolved decisions/dependencies and downstream consequence.",
156 + "decompose only when it reduces decisions, context, or error propagation more than handoff overhead.",
157 + "credit structured upstream judgement; verify when risk warrants.",
158 + "manage context windows: minimum sufficient context; progressive disclosure.",
159 + "choose least fitting tier; lower wins ties; add verification when one child is insufficient.",
160 + "ask user about missing decisions; encode missing child evidence/authority as a blocker; never guess with higher tier.",
159 161 ...Object.entries(tiers).map(
160 162 ([name, tier]) =>
161 163 `- ${name}: ${tier.model ?? "session model"}; ${tier.thinkingLevel}; ${tier.instructions}`,

extensions/ultra/spec.ts

Modified
15 15 const INTERPOLATION_GRAMMAR =
16 16 "Closed grammar: {args.NAME}, {item}, {PHASE.results}, {PHASE.failures}, " +
17 - "{PHASE.results[].FIELD[]}, or {EXPR | where FIELD}.";
17 + '{PHASE.results[].FIELD[]}, {EXPR | where FIELD}, {EXPR | where FIELD == "VALUE"}, or {EXPR | groupBy FIELD}.';
18 18 const SELECTOR_GUIDANCE = `Use exactly one single brace-wrapped selector. ${INTERPOLATION_GRAMMAR}`;
20 20 export const DYNAMIC_EXTENSION_TOOL_NAMES = [
232 232 description: `${SELECTOR_GUIDANCE} Use args or any completed phase results/failures; the selected value may have any type.`,
235 + report: Type.Optional(
236 + Type.String({
237 + description: `${SELECTOR_GUIDANCE} Selects the human-facing report; the selected value may have any type.`,
238 + }),
239 + ),
237 242 description:
248 253 export type Phase = Type.Static<typeof PhaseSchema>;
249 254 export type WorkflowSpec = Type.Static<typeof WorkflowSpecSchema>;
251 -// Authoring details load through the skill, not every provider request.
252 -// parseWorkflow still validates every invocation with the full runtime schema.
253 -export const WorkflowToolSpecSchema = Type.Unsafe<WorkflowSpec>(
254 - Type.Object(
255 - {},
256 - { additionalProperties: true, description: "Inline workflow spec." },
257 - ),
258 -);
260 256 export function isDynamicExtensionBuilderStep(step: Step): boolean {
261 257 const tools = new Set(step.tools ?? []);
262 258 return DYNAMIC_EXTENSION_TOOL_NAMES.every((name) => tools.has(name));

extensions/ultra/wire.ts

Modified
45 45 : dropped > 0
46 46 ? "incomplete"
47 47 : "finished";
48 - const { phaseResults: _phaseResults, phaseFailures, ...output } = result;
48 + const {
49 + phaseResults: _phaseResults,
50 + phaseFailures,
51 + report: _report,
52 + ...output
53 + } = result;
49 54 const text = JSON.stringify(
51 56 ...output,

extensions/ultra/workflows/research.json

Modified
304 - "return": "{report.results}"
304 + "return": "{report.results}",
305 + "report": "{report.results}"