repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

fix(ultra): guide valid inline workflows

190832021dc8bfa271b69c30f2d8f6ca0a4ede04

Oliver Jan Krylow <oliver@bugabinga.net> ·

unsigned

Browse filesCompare with HEAD

Changed files (12)

docs/super/plans/2026-07-17-ultra-tool-guidance.md

Added
new file mode 100644
1 +# Ultra Tool Guidance Implementation Plan
3 +> **For agentic workers:** REQUIRED SUB-SKILL: Use /skill:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
5 +**Goal:** Give `workflow` callers the semantic contract needed to produce valid inline specs on the first attempt.
7 +**Architecture:** Put invocation-critical guidance on the existing shared TypeBox schema so Pi delivers it with the tool contract. Add only two runtime constraints already required by interpolation—identifier-shaped, unique phase IDs—and synchronize the tool reminder, skill, canonical reference, README, and lockstep tests.
9 +**Tech Stack:** TypeScript, TypeBox, Vitest, Pi extension harness, Markdown
11 +**Roadmap:** None
13 +**Phase:** Single-plan implementation
15 +---
17 +## Task 1: Validate phase identifiers
19 +**Files:**
21 +- Modify: `extensions/ultra/__tests__/spec.test.ts`
22 +- Modify: `extensions/ultra/spec.ts`
24 +- [x] **Step 1: Write failing phase-ID tests**
26 +Add tests proving `parseWorkflow` rejects a hyphenated phase ID, reserved ID `args`, and duplicate phase IDs, while allowing `item`:
28 +```ts
29 +it("requires interpolation-safe phase identifiers", () => {
30 + const s = validSpec();
31 + s.phases[0].id = "first-pass";
32 + expect(() => parseWorkflow(s)).toThrow(/id/i);
33 +});
35 +it("rejects duplicate phase identifiers", () => {
36 + const s = validSpec();
37 + s.phases[1].id = s.phases[0].id;
38 + expect(() => parseWorkflow(s)).toThrow(/duplicate phase id.*review/i);
39 +});
41 +it("reserves args but allows item as a phase identifier", () => {
42 + const reserved = validSpec();
43 + reserved.phases[0].id = "args";
44 + expect(() => parseWorkflow(reserved)).toThrow(/reserved phase id.*args/i);
46 + const allowed = validSpec();
47 + allowed.phases[0].id = "item";
48 + expect(() => parseWorkflow(allowed)).not.toThrow();
49 +});
50 +```
52 +- [x] **Step 2: Verify RED**
54 +Run: `mise run test -- extensions/ultra/__tests__/spec.test.ts`
55 +Expected: the hyphenated, reserved, and duplicate-ID assertions fail because all are currently accepted.
57 +- [x] **Step 3: Implement minimal validation**
59 +Define one phase identifier schema using `^[A-Za-z_][A-Za-z0-9_]*$` and use it for both phase variants.
60 +After structural validation, reject `args` with `Invalid workflow spec: reserved phase id "args".` and reject repeated phase IDs with `Invalid workflow spec: duplicate phase id "<id>".`
61 +Do not reserve `item`.
63 +- [x] **Step 4: Verify GREEN**
65 +Run: `mise run test -- extensions/ultra/__tests__/spec.test.ts`
66 +Expected: PASS.
68 +## Task 2: Expose semantic guidance through the tool schema
70 +**Files:**
72 +- Modify: `extensions/ultra/__tests__/skill.test.ts`
73 +- Modify: `extensions/ultra/__tests__/harness.test.ts`
74 +- Modify: `extensions/ultra/spec.ts`
75 +- Modify: `extensions/ultra/index.ts`
77 +- [x] **Step 1: Write failing lockstep tests**
79 +In `skill.test.ts`, serialize `WorkflowSpecSchema` and assert that its descriptions contain all critical contracts already present in the skill/reference:
81 +```ts
82 +const toolGuidance = JSON.stringify(WorkflowSpecSchema);
83 +for (
84 + const rule of [
85 + "exact key in top-level schemas",
86 + "{PHASE.results[].FIELD[]}",
87 + "single brace-wrapped selector",
88 + "read, grep, find, ls, bash",
89 + "forbid mutation",
90 + ]
91 +) {
92 + expect(toolGuidance).toContain(rule);
94 +```
96 +In `harness.test.ts`, assert the registered tool description includes `schema strings must exactly name top-level schemas` and `interpolation uses only the closed grammar`.
98 +- [x] **Step 2: Verify RED**
100 +Run: `mise run test -- extensions/ultra/__tests__/skill.test.ts extensions/ultra/__tests__/harness.test.ts`
101 +Expected: FAIL because schema descriptions, compact tool reminders, and `max` documentation are missing.
103 +- [x] **Step 3: Add shared TypeBox descriptions**
105 +Add concise descriptions to the existing schemas for:
107 +- workflow `name`, `description`, inputs, named schemas, ordered phases, and `return`;
108 +- phase `id`, `kind`, `when`, `over`, `step`, and concurrency;
109 +- step prompt, tools, model, thinking level, and output schema;
110 +- the string and object branches of `step.schema`;
111 +- the literal-array and selector branches of fan-out `over`.
113 +Descriptions must encode the approved selector matrix exactly:
115 +```text
116 +prompt: embedded grammar tokens; item only in fan-out; literal braces forbidden
117 +when: one selector over args/prior phases; false/null/undefined/empty string/empty array skips
118 +over: literal array or one selector over args/prior phases; result must be an array
119 +return: one selector over args or completed phase results/failures; any result type
120 +```
122 +State the closed token forms verbatim and the read-only fan-out tool rule verbatim.
124 +- [x] **Step 4: Add compact top-level tool guidance**
126 +Extend the `workflow` description with two short sentences:
128 +```text
129 +For inline specs, schema strings must exactly name top-level schemas. Interpolation uses only the closed grammar described on the spec fields.
130 +```
132 +Describe tool `args` as values exposed through `{args.NAME}` and keep `background` unchanged.
133 +The shared `WorkflowSpecSchema` itself supplies the inline `spec` description.
135 +- [x] **Step 5: Verify GREEN**
137 +Run: `mise run test -- extensions/ultra/__tests__/skill.test.ts extensions/ultra/__tests__/harness.test.ts`
138 +Expected: PASS.
140 +## Task 3: Synchronize `max` thinking guidance
142 +**Files:**
144 +- Modify: `extensions/ultra/skills/authoring-workflows/SKILL.md`
145 +- Modify: `extensions/ultra/skills/authoring-workflows/references/spec-format.md`
146 +- Modify: `extensions/ultra/README.md`
147 +- Modify: `extensions/ultra/runner.ts`
148 +- Modify: `extensions/ultra/__tests__/runner.test.ts`
149 +- Modify: `extensions/ultra/__tests__/spec.test.ts`
150 +- Modify: `extensions/ultra/__tests__/skill.test.ts`
152 +- [x] **Step 1: Update all guidance surfaces**
154 +Add `max` after `xhigh` everywhere the supported thinking-level set or model-tier suffix is documented.
155 +Describe `max` as the highest opt-in reasoning budget for models that support it.
156 +Keep existing role-based tier guidance unchanged.
158 +- [x] **Step 2: Write and verify the failing tier-suffix test**
160 +Add a runner test using `modelTiers: { small: "openai/gpt-5-mini:max" }` and assert the session receives `thinkingLevel: "max"`.
161 +Run: `mise run test -- extensions/ultra/__tests__/runner.test.ts`
162 +Expected: FAIL because the current suffix parser treats `:max` as part of the model ID.
164 +- [x] **Step 3: Update runtime and complete-example validation**
166 +Add `max` to `splitTierSpec` in `runner.ts`.
167 +Update `spec.test.ts`'s complete thinking-level loop to include `max`.
169 +Change the complete-spec thinking-level assertion to accept:
171 +```ts
172 +/^(off|minimal|low|medium|high|xhigh|max)$/;
173 +```
175 +Add a lockstep assertion that runtime schema, skill, reference, and README all mention `max`.
177 +- [x] **Step 4: Run focused tests**
179 +Run: `mise run test -- extensions/ultra/__tests__/skill.test.ts extensions/ultra/__tests__/spec.test.ts extensions/ultra/__tests__/harness.test.ts extensions/ultra/__tests__/runner.test.ts`
180 +Expected: PASS.
182 +## Task 4: Verify and commit
184 +**Files:**
186 +- Modify: `docs/super/plans/2026-07-17-ultra-tool-guidance.md`
188 +- [x] **Step 1: Mark completed plan steps**
190 +Change every completed checkbox in this plan from `[ ]` to `[x]`.
192 +- [x] **Step 2: Run repository verification sequentially**
194 +Run:
196 +```text
197 +mise run fmt
198 +mise run lint
199 +mise run test -- extensions/ultra
200 +mise run check
201 +```
203 +Expected: formatting makes no unexpected semantic changes; lint reports zero errors; all Ultra tests pass; typecheck and static checks pass.
205 +- [x] **Step 3: Inspect isolation**
207 +Run: `git status --short --branch`
208 +Expected: only Ultra guidance implementation files and this plan are changed in `.worktrees/ultra-tool-guidance`; the original `fix/leaks-reliability` worktree remains untouched.
210 +- [x] **Step 4: Commit**
212 +```text
213 +git add docs/super/plans/2026-07-17-ultra-tool-guidance.md docs/super/specs/2026-07-17-ultra-tool-guidance-design.md extensions/ultra/AGENTS.md extensions/ultra/spec.ts extensions/ultra/index.ts extensions/ultra/runner.ts extensions/ultra/README.md extensions/ultra/skills/authoring-workflows/SKILL.md extensions/ultra/skills/authoring-workflows/references/spec-format.md extensions/ultra/__tests__/spec.test.ts extensions/ultra/__tests__/skill.test.ts extensions/ultra/__tests__/harness.test.ts extensions/ultra/__tests__/runner.test.ts
214 +git commit -m "fix(ultra): guide valid inline workflows"
215 +```

docs/super/specs/2026-07-17-ultra-tool-guidance-design.md

Modified
23 23 Do not auto-repair malformed specs or change runtime error handling because the observed errors already enabled immediate correction.
25 25 Use the identifier pattern already enforced for argument names on phase IDs, matching the interpolation parser's accepted phase syntax.
26 -Reject duplicate phase IDs during `parseWorkflow`; `args` and `item` remain valid phase IDs because interpolation namespaces are segregated.
26 +Reject duplicate phase IDs during `parseWorkflow` and reserve `args`, because `{args.NAME}` takes precedence over phase-result parsing and would make an `args` phase unreferenceable.
27 +`item` remains valid because `{item.results}` is unambiguous.
27 28 These checks prevent unreferenceable IDs and result-map overwrites.
29 30 ## Selector Context Matrix
31 -| Field | Accepted form | Available values | Additional rule |
32 -| --- | --- | --- | --- |
33 -| `step.prompt` | Text containing zero or more grammar tokens | args, current fan-out item, prior phase results/failures, flattened fields, one `where` filter | `{item}` is meaningful only in fan-out; literal braces are forbidden. |
34 -| `phase.when` | Exactly one brace-wrapped selector | args or prior phase results/failures; no current item | False, null, undefined, empty string, or empty array skips. |
35 -| fan-out `over` | Literal array or exactly one brace-wrapped selector | args or prior phase results/failures; no current item | Resolved selector must produce an array. |
36 -| workflow `return` | Exactly one brace-wrapped selector | args or any completed phase results/failures | No required result type. |
32 +| Field | Accepted form | Available values | Additional rule |
33 +| ----------------- | --------------------------------------------------- | ---------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
34 +| `step.prompt` | Text containing zero or more grammar tokens | args, current fan-out item, prior phase results/failures, flattened fields, one `where` filter | `{item}` is meaningful only in fan-out; literal braces are forbidden. |
35 +| `phase.when` | Exactly one brace-wrapped selector | args or prior phase results/failures; no current item | False, null, undefined, empty string, or empty array skips. |
36 +| fan-out `over` | Literal array or exactly one brace-wrapped selector | args or prior phase results/failures; no current item | Resolved selector must produce an array. |
37 +| workflow `return` | Exactly one brace-wrapped selector | args or any completed phase results/failures | No required result type. |
38 39 The closed tokens are `{args.NAME}`, `{item}`, `{PHASE.results}`, `{PHASE.failures}`, `{PHASE.results[].FIELD[]}`, and `{EXPR | where FIELD}`.
39 -Phase and field identifiers use `[A-Za-z_][A-Za-z0-9_]*`.
40 +Phase and field identifiers use `[A-Za-z_][A-Za-z0-9_]*`; phase ID `args` is reserved.
41 42 ## Thinking Levels
43 44 Support Pi's complete current set: `off`, `minimal`, `low`, `medium`, `high`, `xhigh`, and `max`.
44 -Update the bundled skill, canonical reference, README, tool schema guidance, and lockstep tests where they currently stop at `xhigh`.
45 +Update the runtime model-tier suffix parser, bundled skill, canonical reference, README, tool schema guidance, and lockstep tests where they currently stop at `xhigh`.
46 47 ## Semantic Synchronization
53 54 - `extensions/ultra/spec.ts`: add shared schema descriptions, phase-ID pattern, and duplicate-ID rejection.
54 55 - `extensions/ultra/index.ts`: add concise tool-level invocation guidance and run-argument descriptions.
55 -- `extensions/ultra/__tests__/spec.test.ts`: cover phase-ID syntax and uniqueness.
56 +- `extensions/ultra/runner.ts`: accept `max` in configured model-tier suffixes.
57 +- `extensions/ultra/__tests__/spec.test.ts`: cover phase-ID syntax, reserved names, uniqueness, and all thinking levels.
58 +- `extensions/ultra/__tests__/runner.test.ts`: cover the `max` model-tier suffix.
56 59 - `extensions/ultra/__tests__/skill.test.ts`: enforce semantic guidance lockstep.
57 60 - `extensions/ultra/__tests__/harness.test.ts`: verify registered tool guidance reaches callers.
58 61 - `extensions/ultra/skills/authoring-workflows/SKILL.md`, its canonical reference, and `extensions/ultra/README.md`: synchronize current thinking levels and invocation semantics.

extensions/ultra/README.md

Modified
102 102 `ultra.modelTiers` maps workflow step `model` values of `small`, `medium`, or `big` to concrete model refs.
103 103 Unset tiers fall back to the session model.
104 -A tier value may end with a thinking suffix (`:off`, `:minimal`, `:low`, `:medium`, `:high`, `:xhigh`), used when the step omits `thinkingLevel`.
104 +A tier value may end with a thinking suffix (`:off`, `:minimal`, `:low`, `:medium`, `:high`, `:xhigh`, `:max`), used when the step omits `thinkingLevel`; `max` is the highest opt-in budget and requires model support.
105 105 Bundled workflows explicitly assign tiers by phase role and thinking levels by relative complexity.
107 107 ```json

extensions/ultra/__tests__/harness.test.ts

Modified
38 38 const tool = extension.tools.get("workflow").definition;
39 39 const specSchema = tool.parameters.properties.spec;
41 + expect(tool.description).toContain(
42 + "schema strings must exactly name top-level schemas",
43 + );
44 + expect(tool.description).toContain(
45 + "interpolation uses only the closed grammar",
46 + );
41 47 expect(JSON.parse(JSON.stringify(specSchema))).toEqual(
42 48 JSON.parse(JSON.stringify(WorkflowSpecSchema)),

extensions/ultra/__tests__/runner.test.ts

Modified
1076 1076 expect(overrideHarness.captured.args?.thinkingLevel).toBe("high");
1079 + it("applies the max tier thinking suffix", async () => {
1080 + const harness = makeSessionHarness({
1081 + drive: happyDrive({ verdict: "maximum" }),
1082 + });
1084 + await runStep(
1085 + { ...toolStep, model: "small" },
1086 + makeCtx(),
1087 + makeDrivers({ sessionFactory: harness.factory }),
1088 + baseOpts({ modelTiers: { small: "openai/gpt-5-mini:max" } }),
1089 + );
1091 + expect(harness.captured.args?.thinkingLevel).toBe("max");
1092 + });
1079 1094 it("routes a tool-using call through runStep and returns its result", async () => {
1080 1095 const harness = makeSessionHarness({
1081 1096 drive: happyDrive({ verdict: "routed" }),

extensions/ultra/__tests__/skill.test.ts

Modified
2 2 import * as fs from "node:fs";
3 3 import { fileURLToPath } from "node:url";
4 4 import { loadSkillsFromDir } from "@earendil-works/pi-coding-agent";
5 5 import { describe, expect, it } from "vitest";
6 -import { parseWorkflow } from "../spec.ts";
6 +import { parseWorkflow, WorkflowSpecSchema } from "../spec.ts";
8 8 // Root `package.json` registers ultra's skills under `pi.skills` with the
9 9 // explicit entry `extensions/ultra/skills` (alongside other skill-bundling
19 19 import.meta.url,
22 +const README_PATH = fileURLToPath(new URL("../README.md", import.meta.url));
23 24 describe("authoring-workflows companion skill", () => {
24 25 it("bundles exactly one skill subdir — authoring-workflows", () => {
54 55 expect(fs.existsSync(skill.filePath)).toBe(true);
58 + it("exposes invocation-critical semantics through the tool schema", () => {
59 + const toolGuidance = JSON.stringify(WorkflowSpecSchema);
60 + for (const rule of [
61 + "exact key in top-level schemas",
62 + "{PHASE.results[].FIELD[]}",
63 + "single brace-wrapped selector",
64 + "read, grep, find, ls, bash",
65 + "forbid mutation",
66 + ]) {
67 + expect(toolGuidance).toContain(rule);
68 + }
69 + });
71 + it("keeps the exact interpolation grammar synchronized", () => {
72 + const skill = fs.readFileSync(SKILL_PATH, "utf8");
73 + expect(skill).toContain("{EXPR | where FIELD}");
74 + expect(skill).not.toContain("{… | where FIELD}");
75 + });
77 + it("keeps phase identifier guidance synchronized", () => {
78 + expect(JSON.stringify(WorkflowSpecSchema)).toContain(
79 + "reserved identifier args",
80 + );
81 + for (const content of [
82 + fs.readFileSync(SKILL_PATH, "utf8"),
83 + fs.readFileSync(SPEC_FORMAT_PATH, "utf8"),
84 + ]) {
85 + expect(content).toContain("reserved phase ID `args`");
86 + }
87 + });
57 89 it("requires deliberate per-step model tiers and thinking levels", () => {
58 90 const skill = fs.readFileSync(SKILL_PATH, "utf8");
59 91 const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
68 100 expect(reference).toContain("re-evaluate each assignment");
103 + it("keeps max thinking guidance synchronized", () => {
104 + expect(JSON.stringify(WorkflowSpecSchema)).toContain('"const":"max"');
105 + for (const content of [
106 + fs.readFileSync(SKILL_PATH, "utf8"),
107 + fs.readFileSync(SPEC_FORMAT_PATH, "utf8"),
108 + fs.readFileSync(README_PATH, "utf8"),
109 + ]) {
110 + expect(content).toContain("`max`");
112 + });
71 114 it("matches runtime positional and structured-output semantics", () => {
72 115 const skill = fs.readFileSync(SKILL_PATH, "utf8");
73 116 const reference = fs.readFileSync(SPEC_FORMAT_PATH, "utf8");
113 156 for (const phase of spec.phases) {
114 157 expect(phase.step.model).toMatch(/^(small|medium|big)$/);
115 158 expect(phase.step.thinkingLevel).toMatch(
116 - /^(off|minimal|low|medium|high|xhigh)$/,
159 + /^(off|minimal|low|medium|high|xhigh|max)$/,

extensions/ultra/__tests__/spec.test.ts

Modified
112 112 expect(() => parseWorkflow(s)).not.toThrow();
115 + it("requires interpolation-safe phase identifiers", () => {
116 + const s = validSpec();
117 + s.phases[0].id = "first-pass";
118 + expect(() => parseWorkflow(s)).toThrow(/id/i);
119 + });
121 + it("rejects duplicate phase identifiers", () => {
122 + const s = validSpec();
123 + s.phases[1].id = s.phases[0].id;
124 + expect(() => parseWorkflow(s)).toThrow(/duplicate phase id.*review/i);
125 + });
127 + it("reserves args but allows item as a phase identifier", () => {
128 + const reserved = validSpec();
129 + reserved.phases[0].id = "args";
130 + expect(() => parseWorkflow(reserved)).toThrow(/reserved phase id.*args/i);
132 + const allowed = validSpec();
133 + allowed.phases[0].id = "item";
134 + expect(() => parseWorkflow(allowed)).not.toThrow();
135 + });
115 137 it("throws (not returns) on missing `name`, surfacing the TypeBox message", () => {
116 138 const s = validSpec();
117 139 delete s.name;
140 162 expect(parseWorkflow(s).phases[0].step.thinkingLevel).toBe("low");
143 - it("accepts every thinkingLevel in off|minimal|low|medium|high|xhigh", () => {
144 - for (const level of ["off", "minimal", "low", "medium", "high", "xhigh"]) {
165 + it("accepts every thinkingLevel in off|minimal|low|medium|high|xhigh|max", () => {
166 + for (const level of [
167 + "off",
168 + "minimal",
169 + "low",
170 + "medium",
171 + "high",
172 + "xhigh",
173 + "max",
174 + ]) {
145 175 const s = validSpec();
146 176 s.phases[0].step.thinkingLevel = level;
147 177 expect(() => parseWorkflow(s)).not.toThrow();

extensions/ultra/index.ts

Modified
254 254 "concurrency cap and return aggregated JSON. Do not use it for an ordinary single-agent task; " +
255 255 "use one agent only when isolation, background execution, an alternate model, resumability, or " +
256 256 "strict structured output is specifically needed. Pass either a saved workflow `name` " +
257 - "(+ optional `args`), or an inline `spec` object for a one-off run.",
257 + "(+ optional `args`), or an inline `spec` object for a one-off run. For inline specs, " +
258 + "schema strings must exactly name top-level schemas; interpolation uses only the closed grammar " +
259 + "described on the spec fields.",
258 260 parameters: Type.Object({
259 261 name: Type.Optional(
260 262 Type.String({
264 266 spec: Type.Optional(WorkflowSpecSchema),
265 - args: Type.Optional(Type.Record(Type.String(), Type.Unknown())),
267 + args: Type.Optional(
268 + Type.Record(Type.String(), Type.Unknown(), {
269 + description:
270 + "Workflow input values exposed to prompts and selectors as {args.NAME}.",
271 + }),
272 + ),
266 273 background: Type.Optional(
267 274 Type.Boolean({
268 275 description: "Start the workflow in the background and report later.",

extensions/ultra/runner.ts

Modified
262 262 model: string;
263 263 thinkingLevel?: ThinkingLevel;
265 - const match = spec.match(/^(.*):(off|minimal|low|medium|high|xhigh)$/);
265 + const match = spec.match(/^(.*):(off|minimal|low|medium|high|xhigh|max)$/);
266 266 if (!match) return { model: spec };
267 267 return { model: match[1], thinkingLevel: match[2] as ThinkingLevel };

extensions/ultra/skills/authoring-workflows/SKILL.md

Modified
43 43 judges or merges. Sketch the phases before writing JSON.
45 45 2. **Co-design the phases.** For each phase decide:
46 + - Give every phase a unique interpolation identifier; the reserved phase ID `args` is forbidden.
46 47 - `single` (one synthesis/judge pass) vs. `fanout` (one agent per item).
47 48 - `fanout` needs an `over` — a literal array, or an interpolation token over
48 49 a prior phase's results (e.g. `{review.results[].findings[]}`).
61 62 worked example in the reference.
63 64 5. **Wire interpolation.** Use only the closed grammar (`{args.NAME}`, `{item}`,
64 - `{PHASE.results}`, `{PHASE.failures}`, `{PHASE.results[].FIELD[]}`, `{… | where FIELD}`). Keep
65 + `{PHASE.results}`, `{PHASE.failures}`, `{PHASE.results[].FIELD[]}`, `{EXPR | where FIELD}`). Keep
65 66 literal `{`/`}` out of prompts except these tokens. Set `return` to the
66 67 selector for the workflow's final value.
69 70 - `big` — planning, specification, architecture, critique, synthesis.
70 71 - `medium` — coding, testing, editing, routine implementation.
71 72 - `small` — scouting, extraction, classification, easy text work.
72 - - Set `thinkingLevel` from that step's relative ambiguity, constraints, and cost of mistakes.
73 + - Set `thinkingLevel` from `off`, `minimal`, `low`, `medium`, `high`, `xhigh`, or `max` based on that step's relative ambiguity, constraints, and cost of mistakes. `max` is the highest opt-in budget and requires model support.
73 74 - Explicitly set both fields. If every phase uses the same pair, challenge the assignment before saving.
75 76 7. **Self-check the spec (there is no validator tool).** You can't call

extensions/ultra/skills/authoring-workflows/references/spec-format.md

Modified
24 24 A phase has an `id` (referenced by interpolation and `return`), a `kind`, a
25 25 `step`, an optional `concurrency` cap, and an optional `when` selector.
26 +Phase IDs must be unique identifiers matching `[A-Za-z_][A-Za-z0-9_]*`; the reserved phase ID `args` is forbidden because it belongs to the workflow-input namespace.
27 28 - **`single`** — runs the step exactly once, for one synthesis / judge /
28 29 aggregation pass over prior results. (`over` is accepted by the schema but
54 55 | `prompt` | yes | string | The agent instruction. Interpolates `{...}` tokens (see below). |
55 56 | `tools` | no | string[] | Tool allowlist. Omit for a tool-less completion (pure reasoning). |
56 57 | `model` | no | string | `provider/id` override, or configured tier `small`/`medium`/`big`; defaults to the session model. |
57 -| `thinkingLevel` | no | off…xhigh | Reasoning budget for this step. |
58 +| `thinkingLevel` | no | off…max | Reasoning budget for this step. |
58 59 | `schema` | no | string \| JSON Schema | A `schemas` name, or an inline schema that validates structured output. |
60 61 Every step returns a JSON object through completion parsing or the terminating `structured_output` tool.
85 86 - `off` / `minimal` — deterministic extraction, transcription, or formatting with negligible ambiguity.
86 87 - `low` / `medium` — routine decisions with bounded context and cheap mistakes and when context has been prepared beforehand.
87 -- `high` / `xhigh` — ambiguous, multi-constraint reasoning where a wrong decision is costly downstream.
88 +- `high` / `xhigh` / `max` — ambiguous, multi-constraint reasoning where a wrong decision is costly downstream. `max` is the highest opt-in budget and requires model support.
89 90 If every phase has the same model/thinking pair, re-evaluate each assignment before saving.
90 91 Uniform settings are valid only when the phases genuinely perform equally difficult work of the same kind.

extensions/ultra/spec.ts

Modified
10 10 // Schemas (TypeBox -> JSON Schema fragments)
11 11 // ---------------------------------------------------------------------------
13 +const INTERPOLATION_GRAMMAR =
14 + "Closed grammar: {args.NAME}, {item}, {PHASE.results}, {PHASE.failures}, " +
15 + "{PHASE.results[].FIELD[]}, or {EXPR | where FIELD}.";
16 +const SELECTOR_GUIDANCE = `Use exactly one single brace-wrapped selector. ${INTERPOLATION_GRAMMAR}`;
13 18 /** Any object accepted as an inline JSON Schema (TypeBox-compatible). */
14 -const JsonSchemaObject = Type.Object({}, { additionalProperties: true });
19 +const JsonSchemaObject = Type.Object(
20 + {},
21 + {
22 + additionalProperties: true,
23 + description: "An inline JSON Schema object for this step's JSON output.",
24 + },
25 +);
16 27 /** Full `ThinkingLevel` union. */
17 -export const ThinkingLevelSchema = Type.Union([
18 - Type.Literal("off"),
19 - Type.Literal("minimal"),
20 - Type.Literal("low"),
21 - Type.Literal("medium"),
22 - Type.Literal("high"),
23 - Type.Literal("xhigh"),
24 - Type.Literal("max"),
25 -]);
28 +export const ThinkingLevelSchema = Type.Union(
29 + [
30 + Type.Literal("off"),
31 + Type.Literal("minimal"),
32 + Type.Literal("low"),
33 + Type.Literal("medium"),
34 + Type.Literal("high"),
35 + Type.Literal("xhigh"),
36 + Type.Literal("max"),
37 + ],
38 + {
39 + description:
40 + "Reasoning budget: off, minimal, low, medium, high, xhigh, or max. " +
41 + "max is the highest opt-in level and requires model support.",
42 + },
43 +);
27 45 /** A step's `schema` is either a named reference (string) or an inline object. */
28 -const StepSchemaRef = Type.Union([Type.String(), JsonSchemaObject]);
46 +const StepSchemaRef = Type.Union(
47 + [
48 + Type.String({
49 + description:
50 + "A named output schema reference: use the exact key in top-level schemas, not a JSON Schema type or label.",
51 + }),
52 + JsonSchemaObject,
53 + ],
54 + {
55 + description:
56 + "Constrains the JSON object returned by this step. Use an exact named reference or an inline JSON Schema object.",
57 + },
58 +);
30 60 export const StepSchema = Type.Object(
32 - prompt: Type.String(),
33 - tools: Type.Optional(Type.Array(Type.String())),
34 - model: Type.Optional(Type.String()),
62 + prompt: Type.String({
63 + description:
64 + `Sub-agent instruction. May embed grammar tokens in text; {item} is meaningful only in fanout. ${INTERPOLATION_GRAMMAR} ` +
65 + "Do not use literal braces for any other purpose.",
66 + }),
67 + tools: Type.Optional(
68 + Type.Array(Type.String(), {
69 + description:
70 + "Sub-agent tool allowlist. Fanout may use only read, grep, find, ls, bash, and its prompt must forbid mutation. Put writing tools in a single phase.",
71 + }),
72 + ),
73 + model: Type.Optional(
74 + Type.String({
75 + description:
76 + "Model provider/id or configured tier small, medium, or big. Set deliberately for every authored step.",
77 + }),
78 + ),
35 79 thinkingLevel: Type.Optional(ThinkingLevelSchema),
36 80 schema: Type.Optional(StepSchemaRef),
38 - { additionalProperties: false },
82 + {
83 + additionalProperties: false,
84 + description:
85 + "One sub-agent invocation. Every step must return a JSON object; schema makes its output shape stable.",
86 + },
41 89 /** `over` accepts a literal array or an interpolation expression string. */
42 -const OverSchema = Type.Union([Type.Array(Type.Unknown()), Type.String()]);
90 +const OverSchema = Type.Union(
91 + [
92 + Type.Array(Type.Unknown(), {
93 + description: "Literal items for one fanout agent per array element.",
94 + }),
95 + Type.String({
96 + description: `${SELECTOR_GUIDANCE} Use args or prior phase results/failures; the resolved value must be an array.`,
97 + }),
98 + ],
99 + {
100 + description:
101 + "Fanout input: a literal array or one selector that resolves to an array.",
102 + },
105 +const PhaseIdSchema = Type.String({
106 + pattern: "^[A-Za-z_][A-Za-z0-9_]*$",
107 + description:
108 + "Unique interpolation identifier for this phase. The reserved identifier args is forbidden.",
109 +});
111 +const WhenSchema = Type.Optional(
112 + Type.String({
113 + description: `${SELECTOR_GUIDANCE} Use args or prior phase results/failures; false, null, undefined, an empty string, or an empty array skips the phase.`,
114 + }),
116 +const ConcurrencySchema = Type.Optional(
117 + Type.Number({
118 + description: "Optional maximum concurrent agents for this phase.",
119 + }),
44 122 // Phase is a discriminated union so `over` is *structurally* required for
45 123 // `fanout` and absent-tolerant for `single` — a fanout without `over` is
46 124 // rejected by TypeBox itself, no extra semantic pass needed.
47 -const SinglePhaseSchema = Type.Object({
48 - id: Type.String(),
49 - kind: Type.Literal("single"),
50 - when: Type.Optional(Type.String()),
51 - over: Type.Optional(OverSchema),
52 - step: StepSchema,
53 - concurrency: Type.Optional(Type.Number()),
125 +const SinglePhaseSchema = Type.Object(
127 + id: PhaseIdSchema,
128 + kind: Type.Literal("single", {
129 + description:
130 + "Run one synthesis, judge, or writing step. Any over value is ignored.",
131 + }),
132 + when: WhenSchema,
133 + over: Type.Optional(OverSchema),
134 + step: StepSchema,
135 + concurrency: ConcurrencySchema,
136 + },
137 + { description: "A phase that invokes one sub-agent." },
140 +const FanoutPhaseSchema = Type.Object(
142 + id: PhaseIdSchema,
143 + kind: Type.Literal("fanout", {
144 + description: "Run one read-only sub-agent per over item.",
145 + }),
146 + when: WhenSchema,
147 + over: OverSchema,
148 + step: StepSchema,
149 + concurrency: ConcurrencySchema,
150 + },
151 + { description: "A phase that invokes parallel read-only sub-agents." },
154 +export const PhaseSchema = Type.Union([SinglePhaseSchema, FanoutPhaseSchema], {
155 + description: "An ordered single or fanout workflow phase.",
56 -const FanoutPhaseSchema = Type.Object({
57 - id: Type.String(),
58 - kind: Type.Literal("fanout"),
59 - when: Type.Optional(Type.String()),
60 - over: OverSchema,
61 - step: StepSchema,
62 - concurrency: Type.Optional(Type.Number()),
63 -});
65 -export const PhaseSchema = Type.Union([SinglePhaseSchema, FanoutPhaseSchema]);
67 -export const WorkflowSpecSchema = Type.Object({
68 - name: Type.String({ minLength: 1 }),
69 - description: Type.Optional(Type.String()),
70 - args: Type.Optional(
71 - Type.Object(
72 - {
73 - hint: Type.Optional(Type.String()),
74 - // Declared positional params: a bare `/ultra <name> <tok...>` line
75 - // maps whitespace tokens to these names in order, exposed as
76 - // `{args.NAME}`. Names must be interpolation identifiers.
77 - params: Type.Optional(
78 - Type.Array(Type.String({ pattern: "^[A-Za-z_][A-Za-z0-9_]*$" })),
79 - ),
80 - },
81 - { additionalProperties: true },
158 +export const WorkflowSpecSchema = Type.Object(
160 + name: Type.String({
161 + minLength: 1,
162 + description: "Non-empty workflow identifier.",
163 + }),
164 + description: Type.Optional(
165 + Type.String({ description: "One-line workflow purpose." }),
83 - ),
84 - schemas: Type.Optional(Type.Record(Type.String(), JsonSchemaObject)),
85 - phases: Type.Array(PhaseSchema, { minItems: 1 }),
86 - return: Type.Optional(Type.String()),
87 -});
167 + args: Type.Optional(
168 + Type.Object(
170 + hint: Type.Optional(
171 + Type.String({ description: "Human-facing argument usage hint." }),
172 + ),
173 + params: Type.Optional(
174 + Type.Array(Type.String({ pattern: "^[A-Za-z_][A-Za-z0-9_]*$" }), {
175 + description:
176 + "Ordered positional input names exposed as {args.NAME}; the last name absorbs remaining command text.",
177 + }),
178 + ),
179 + },
181 + additionalProperties: true,
182 + description: "Workflow input declaration.",
183 + },
184 + ),
185 + ),
186 + schemas: Type.Optional(
187 + Type.Record(Type.String(), JsonSchemaObject, {
188 + description:
189 + "Named JSON output schemas. Every string step.schema must exactly match one key here.",
190 + }),
191 + ),
192 + phases: Type.Array(PhaseSchema, {
193 + minItems: 1,
194 + description:
195 + "Ordered phases. Prompts and selectors may reference only prior phases; return runs after all phases.",
196 + }),
197 + return: Type.Optional(
198 + Type.String({
199 + description: `${SELECTOR_GUIDANCE} Use args or any completed phase results/failures; the selected value may have any type.`,
200 + }),
201 + ),
202 + },
204 + description:
205 + "Inline declarative workflow spec. Follow every field description; malformed semantic references fail before agents run.",
206 + },
89 209 // ---------------------------------------------------------------------------
90 210 // Inferred types
146 266 // must resolve against the spec's `schemas` map.
147 267 const spec = value as WorkflowSpec;
148 268 const schemaNames = new Set(Object.keys(spec.schemas ?? {}));
269 + const phaseIds = new Set<string>();
149 270 for (const phase of spec.phases) {
271 + if (phase.id === "args") {
272 + throw new Error('Invalid workflow spec: reserved phase id "args".');
274 + if (phaseIds.has(phase.id)) {
275 + throw new Error(
276 + `Invalid workflow spec: duplicate phase id "${phase.id}".`,
277 + );
279 + phaseIds.add(phase.id);
150 281 const ref = phase.step.schema;
151 282 if (typeof ref === "string" && !schemaNames.has(ref)) {
152 283 const known =