repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

docs(ultra): design actionable failures

b0c85f10fdd5f8f348cd6e9db88e6742f4b94719

Oliver Jan Krylow <oliver@bugabinga.net> ·

unsigned

Browse filesCompare with HEAD

Changed files (1)

docs/super/specs/2026-07-13-ultra-actionable-failures-design.md

Added
new file mode 100644
1 +# Ultra Actionable Failures Design
3 +## Goal
5 +Make every dropped Ultra step actionable for both the outer orchestrator and later linear workflow phases.
7 +Elapsed time never creates a failure.
8 +Ultra still has no timeout or automatic loop.
10 +## Failure contract
12 +A failed runner result carries one typed `StepFailure`:
14 +```ts
15 +interface StepFailure {
16 + code: FailureCode;
17 + message: string;
18 + retryable: boolean;
19 + attempts: number;
20 + model?: string;
21 + thinkingLevel?: ThinkingLevel;
22 + transcriptTail?: string;
24 +```
26 +`FailureCode` is a closed union:
28 +- `auth`
29 +- `transport`
30 +- `invalid-json`
31 +- `schema`
32 +- `missing-output`
33 +- `aborted`
34 +- `session`
36 +`transcriptTail` is capped at 2,000 characters.
37 +It contains the end of the last unusable model response, not a full session transcript.
38 +This gives repair agents useful evidence without flooding the outer context.
40 +Runner results become a discriminated union.
41 +Successful results carry `ok: true`, `value`, and optional usage.
42 +Failed results carry `ok: false`, `value: null`, `failure`, and optional usage.
43 +Free-form top-level `reason` is removed.
45 +## Phase failures
47 +The engine enriches every runner failure with positional workflow context:
49 +```ts
50 +interface PhaseFailure extends StepFailure {
51 + agentId: string;
52 + index: number;
53 + item?: unknown;
55 +```
57 +Unlaunched work after workflow cancellation becomes an `aborted` failure with zero attempts.
58 +A manually aborted running agent becomes an `aborted` failure with one attempt.
60 +The aggregate workflow result adds:
62 +```ts
63 +phaseFailures: Record<string, PhaseFailure[]>;
64 +```
66 +`phaseResults` remains the positional success/failure record and carries the same typed failure on failed slots.
67 +The outer orchestrator receives both structures in the workflow tool’s JSON content.
69 +## Linear recovery
71 +The interpolation grammar adds:
73 +```text
74 +{PHASE.failures}
75 +```
77 +It resolves to that prior phase’s `PhaseFailure[]`.
78 +Existing `| where FIELD` filtering works on it without a new expression language.
80 +A workflow can therefore add one explicit recovery phase:
82 +```json
84 + "id": "recover",
85 + "kind": "fanout",
86 + "when": "{work.failures}",
87 + "over": "{work.failures}",
88 + "step": {
89 + "prompt": "Repair this failed task: {item}"
90 + }
92 +```
94 +This preserves Ultra’s ordered phase engine.
95 +There is no implicit retry policy, loop, back-edge, or state machine.
96 +Workflow authors choose whether and how recovery consumes failures.
98 +## Failure production
100 +Tool-less completions classify terminal failures as follows:
102 +- unavailable model auth → `auth`, not retryable, zero attempts;
103 +- thrown request, absent response, or model error after retries → `transport`, retryable;
104 +- missing or malformed JSON after retries → `invalid-json`, retryable;
105 +- schema mismatch after retries → `schema`, retryable;
106 +- workflow cancellation → `aborted`, retryable.
108 +Tool-using sessions classify terminal failures as follows:
110 +- cancellation → `aborted`, retryable;
111 +- session creation or prompt exception → `session`, retryable;
112 +- normal session end without the structured-output tool → `missing-output`, retryable.
114 +All expected runner failures resolve as typed failures.
115 +Spec/interpolation authoring errors still fail the workflow immediately.
117 +## UI
119 +`dropped` remains a compact visual state rather than a diagnostic schema.
120 +The board continues showing the failure message.
121 +Expanded final output renders the code, retryability, attempts, model, item, and transcript tail.
122 +Collapsed output keeps counts and short reasons only.
124 +## Documentation
126 +The README and workflow-authoring references document:
128 +- the failure object;
129 +- `{PHASE.failures}`;
130 +- an explicit recovery-phase example;
131 +- no timeout and no automatic retries beyond each step’s configured output retries.
133 +## Verification
135 +Tests cover:
137 +- every runner failure classification;
138 +- transcript-tail capping;
139 +- session exceptions becoming failures rather than workflow rejection;
140 +- aborted launched and unlaunched work;
141 +- phase failure enrichment;
142 +- `{PHASE.failures}` in templates, `when`, `over`, and `return`;
143 +- outer tool JSON containing typed diagnostics;
144 +- one bundled fixture workflow recovering failed items in a later linear phase;
145 +- unchanged infinite-runtime behavior.