flowchart LR
  b1["1 Seeded provider faults"] --> b2["2 Invariant oracles"]
  b2 --> b3["3 Tool faults"]
  b2 --> b4["4 HTTP edge faults"]
  b3 --> b5["5 Seed sweep command"]
  b4 --> b5
  b5 --> b6["6 CI sweep"]

flowchart TB
  sched["Fault schedule (seed)"] --> sprov["Simulated StreamFn"]
  sched --> stool["Simulated tool effects"]
  sched --> clock["Injected clock and cancel"]
  sprov --> agent["smith-core agent loop"]
  stool --> agent
  clock --> agent
  agent --> store["Session store and trace"]
  store --> oracle["Invariant oracles"]
  corpus["Pinned seed corpus"] --> sched

---
id: SMH-PLAN-P3EL04N4
type: plan
title: "Deterministic Provider and Tool Fault Simulation"
spec: SMH-SPEC-SPEC0001
status: draft
depends_on: [SMH-PLAN-DFOG846Z]
---

# Deterministic Provider and Tool Fault Simulation

## Intent

Smith's flakiness comes from providers and tools, not the OS.
Simulate their misbehavior deterministically, in the spirit of TigerBeetle's simulator, scaled to Smith's risk.
The core is already a pure function over one ordered event log; faults enter only through that log.

## Decisions

- Faults come from a seeded schedule; same seed, same run, same trace.
- Time is injected; no sleeps, no wall-clock waits, deadlines on every wait.
- A failing seed is a regression: it is checked in and replayed on every run.
- Oracles check invariants, not scripts: no lost durable record, no torn result shown as success, every effect ends in one explicit outcome, cancellation always settles, bounds always hold.
- Real HTTP and process paths are exercised at the edges; the core loop runs against simulated streams.
- New skill `fault-simulation` teaches authors to add a fault kind, an oracle, and a pinned seed.

## Fault catalog

| Source | Faults |
| --- | --- |
| Provider stream | timeout before first byte, stall mid-stream, truncated body, malformed SSE, malformed JSON, 429 and 5xx, auth expiry, oversized body, duplicate or reordered tool-call fragments |
| Tool | nonzero exit, hang past timeout, huge output, binary output, partial output then failure, cancellation mid-effect |
| Harness | cancellation at every chunk boundary, storage write failure during effect recording |

## Behaviors

Ordered by delivery value.

1. A seeded provider fault schedule drives an eval turn; the run settles with an explicit outcome and a replayable trace.
2. Oracles assert durable-effect and cancellation invariants after every simulated turn.
3. Tool faults run through the real tool boundary with bounded output and recorded mutation state.
4. Real HTTP edge: a local scripted server replays provider faults through the shared HTTP engine.
5. Seed sweep: `cargo x sim` runs N seeds; failures print the seed and write it to the regression corpus.
6. Nightly CI runs a larger sweep; push runs the pinned corpus.

```mermaid
flowchart LR
  b1["1 Seeded provider faults"] --> b2["2 Invariant oracles"]
  b2 --> b3["3 Tool faults"]
  b2 --> b4["4 HTTP edge faults"]
  b3 --> b5["5 Seed sweep command"]
  b4 --> b5
  b5 --> b6["6 CI sweep"]
```

## Structure

```mermaid
flowchart TB
  sched["Fault schedule (seed)"] --> sprov["Simulated StreamFn"]
  sched --> stool["Simulated tool effects"]
  sched --> clock["Injected clock and cancel"]
  sprov --> agent["smith-core agent loop"]
  stool --> agent
  clock --> agent
  agent --> store["Session store and trace"]
  store --> oracle["Invariant oracles"]
  corpus["Pinned seed corpus"] --> sched
```

## Touches

- `smith-harness`: fault schedule, simulated provider stream beside `mock_stream`, simulated tool effects, oracles.
- `smith-core`: agent loop, tools, store, trace; injection seams only where missing.
- `smith-harness/src/http.rs`: HTTP edge fault server for behavior 4.
- `xtask`: `sim` command; nightly job in `.ci/nightly.kdl`.
- `.pi/skills/fault-simulation/SKILL.md`: new testing skill.
- `.system/RULES.md` Testing: one line adding simulation as a testing kind; needs human approval.

## Risks

- Seams added for injection must not change production behavior or cost; instruction budgets gate that.
- Oracles too weak prove nothing; each oracle lands with a mutation proving it catches a planted fault.
- Schedule explosion; keep fault kinds few and composable.

## Exit

- [ ] Every fault kind in the catalog runs under a seed and settles with an explicit outcome.
- [ ] Every oracle has a planted-fault test it catches.
- [ ] A failing seed replays byte-identically from the corpus.
- [ ] `cargo x sim` and the nightly sweep run green without retries.