---
id: SMH-PLAN-QL9EGZAT
type: plan
title: "Durable Submissions, Recovery, and Observation"
spec: SMH-SPEC-SPEC0001
status: draft
depends_on: [SMH-PLAN-KFFL0_FT]
---

# Durable Submissions, Recovery, and Observation

## Outcome

Submit once, lose the connection or process, reopen, and inspect the same admitted work without duplicate effects.
Eval, RPC, embedding, and inspection use one bounded durable authority; observers recover committed history independently of transient progress.

## Scope and ownership

Implement Runtime foundation and Sessions → Committed changes, Submission and receipts, Work and recovery, Scoped state, Observation in [Smith Specification](../../specs/SMH-SPEC-SPEC0001-smith-specification/index.md).
Implement shared state/work mechanisms, not a public task-definition engine, second store, document database, or persisted execution stack.
Preserve binary storage, typed core records, bounded CBOR extension payloads, pure core transitions, unknown-effect policy, and existing crate edges.

[Paged storage](../SMH-PLAN-KFFL0_FT-borrowed-codecs/index.md) supplies writer/observer access, binary codecs, bounded reads, inspection, and recovery seams.
Its approved text remains unchanged; this plan owns the newly required atomic semantic-change extension and consumers.
[Recorded context](../SMH-PLAN-99TPLN9I-recorded-context-and-dispatch-provenance/index.md) owns effective request/dispatch content, not a competing commit implementation.
[Plugin authority](../SMH-PLAN-SLXZ2R9V-plugin-tool-authority-and-run-settlement/index.md) owns loadout/grant policy, nested composition, and lifecycle arbitration.
[Plugin resources](../SMH-PLAN-XRJS3NMU-durable-plugin-state-and-recovery-resources/index.md) owns granted state handles and recovered component binding over this plan's records.
[Harness](../SMH-PLAN-PLUG0001-plugin-harness/index.md) owns basic component execution/grants; [provider plugins](../SMH-PLAN-AIAGD002-provider-and-oauth-plugins/index.md) owns vendor/OAuth delivery.
No approved plan is rewritten; existing draft plans consume these shared contracts when their slices reach them.

Existing seams: `smith-core::{session,frame,content_codec,store,tools,agent,trace}`, `smith-harness::{runtime,storage,http,inspection}`, `smith::provider`, `smith-rpc`, `smith-cli`.
Current entry append is not a multi-record commit; tool result and mutation metadata append separately.
Current Agent polls futures and dispatches tools; current WIT is tool-only.
Candidate new modules: core committed-change/work/state projections and shared command/receipt vocabulary, extracted only with their first consumer.

## Delivery dependencies

Solid arrows order local behavior and prerequisite storage; dashed arrows are behavior-level handoffs, not whole-plan completion gates.
Other plan titles identify coordination only; board state is not implementation evidence.

```mermaid
flowchart TB
  store["KFFL0_FT: storage"] --> b1["B1: durable admission"]
  b1 --> b2["B2: recorded turn"]
  b2 --> b3["B3: effect recovery"]
  b3 --> b4["B4: ownership and close"]
  b4 --> b5["B5: bounded observation"]
  b5 --> b6["B6: fork isolation"]
  harness["PLUG0001: component effects"] -.-> b2
  pins["SLXZ2R9V B2: round pin"] -.-> b2
  context["99TPLN9I B3: request snapshot"] -.-> b2
  b1 -.-> context
  b3 -.-> resources["XRJS3NMU: plugin resources"]
  children["SLXZ2R9V B3-B5: child lifecycle"] -.-> b4
```

B1 has no component prerequisite.
B2 consumes ordinary provider/tool components or conformance components through the real loader; neither full vendor rollout nor companion-plan completion is required.
Missing prerequisite contracts block their consumer, not earlier independent slices.
The handoff order is B1 commit API → recorded-context B3; authority B2 and context B3 → B2 here; B3 here → plugin resources.
Do not make harness/bootstrap depend on this entire plan or create a reciprocal whole-plan dependency.

## Behaviors

Each slice integrates required storage, state, host, and public surface together, then leaves existing modes working.
Tests first pin existing behavior before refactoring; no horizontal “build all layers first” sequence.

### B1 · Recover an admitted input after its acknowledgement is lost

Add durable submission identity and caller key to shared host admission, RPC prompt responses, and bounded `sessions get` inspection.
Touch `smith` IDs/errors/commands, core session/frame/content codecs/store, harness storage/runtime/inspection, CLI, RPC.

- Admit input, resolved destination branch, queue placement, and initial work identity as one bounded semantic change.
- Introduce a versioned bounded commit envelope for dependent typed records; choose concrete framing during implementation before altering golden files.
- Recovery, fast-open checkpoints, readers, selection, replay, and observers recognize complete commit boundaries; member frames alone are not publishable transitions.
- Separate complete visibility from durability acknowledgement; persistence failure prevents state adoption, successful receipt, and dependent effect execution.
- Recover a complete change after lost caller acknowledgement by validating and establishing its persistence boundary; incomplete changes have no partial semantic effect.
- Deduplicate by session and caller key using bounded lookup; same input returns original receipt, conflicting content or explicitly changed destination fails.
- Retrying an implicit destination resolves to the original admission, not a newly selected branch; distinguish omitted destination from explicit conflict.
- New fork sessions do not adopt source admission keys; read-only admission fails before mutation.
- Inspect/status reads never schedule work; queued work remains pending until an explicit execution operation consumes it.

Proof: public embedding API and built RPC admit while execution is held by an explicit test barrier; terminate after commit before response, reopen, retry, inspect one identity and one queued input.
Crash/truncation at each group boundary, corrupt member with intact framing, false checkpoint-shaped payloads, oversized input, duplicate concurrent admission, and observer access expose no partial queue/work state.
Retain unknown records losslessly; unsupported required semantics fail instead of authorizing execution.

### B2 · Run the same admitted turn through embedding, eval, and RPC

Route all turn entrypoints through admission/receipt and one recorded-input driver, retaining existing agent policy.
Touch `smith::provider` and shared runtime vocabulary, core agent/tools/work projections, harness runtime/http/storage, CLI eval, RPC worker.

- Replace public futures-driven core polling with synchronous typed transition inputs and bounded host delivery; preserve normalized provider vocabulary.
- Move external execution out of core transitions; stage changes from recorded inputs and adopt only after durable acknowledgement.
- Drive provider and tool phases with typed checkpoints rather than serialized closures, stacks, or a generic task language.
- Acquire effective request and generation/loadout from companion contracts; preserve them through response and complete tool batch.
- Record redacted provider/effect input before dispatch, feed bounded chunks/outcomes through the same authority, and retain domain-specific terminal outcomes.
- Eval's blocking convenience submits then waits; RPC correlates transport requests separately from durable receipt identity.
- Remove replaced duplicate execution/control paths, without expanding into the owning plans' complete vendor or bundled-tool migrations.

Proof: built eval and RPC plus embedding execute one provider round, ordinary component tool effect, and follow-up against a scripted loopback endpoint with real files.
Inspect equal admitted-work, causal effect, and settlement semantics; no native tool-name fallback or fabricated generation ID counts as parity.
Record failures before dispatch produce no endpoint request or file change; core transition tests replay injected events without OS I/O.

### B3 · Recover an interrupted effect without duplicating it

Make effect result, mutation evidence, dependent state/checkpoint, and receipt changes atomic over B1.
Touch core tools/work/state/content/store/trace, harness effect dispatch/storage, shared effect errors and inspection output.

- Persist intent before execution; poison further mutations/dispatch on uncertain persistence until explicit recovery.
- Atomically commit bounded completion output, outcome, three-state mutation evidence, and work transition.
- Reconstruct pending/checkpoint/terminal work independently of volatile handles; committed completion never runs again during reconstruction.
- Missing or non-authoritative completion remains unknown; block effect-dependent continuation rather than automatically retrying it.
- Show required reconciliation and best-known evidence in receipt inspection; no generic automatic reconciliation feature is introduced.
- Persist execution provenance required for continuation; unavailable code/grants report blocked prerequisites, not successful completion or same-name replacement.
- Prove state/checkpoint atomicity using core-owned work state now; expose the same validated mechanism to the plugin-resource companion, not another store.

Proof: ordinary component writes a real file once; kill at intent, during effect, after effect before completion, and after durable completion before reply.
Reopen/inspect never writes again; completion reconstructs once, unknown remains visible, applied-but-unrecorded failure reports known evidence outside the failed record.
That out-of-record evidence is diagnostic-only: it cannot stand in for committed completion, unblock dependent work, or authorize successful settlement.
Only an explicit recorded effect-specific reconciliation can resolve unknown evidence; this plan introduces no generic reconciliation engine.
Injected storage rejection and uncertain sync failure distinguish guaranteed no-change from recovery-required state.
Use explicit readiness/release/crash hooks in test orchestration, not sleeps or production bypass controls.

### B4 · Abort or close without losing owned work

Wire durable ownership, cancellation intent, waiting, terminal receipts, and close into existing shared driver.
Touch core work/queue/agent, harness orchestration, RPC wait/abort/status, shared cancel/receipt vocabulary.

- Keep wait cancellation local to the waiter; authorized abort records intent and signals execution without promising non-application.
- Parent receipt becomes terminal only after owned work is terminal; compose companion child-lifecycle decisions rather than inventing another handler dispatcher.
- Preserve queued input unless an accepted queue action withdraws it; receipt outcome and mutation evidence stay independent.
- Bound live-work count and owner-chain depth at admission, including repeated admissions across calls; reject excess atomically.
- Close seals admission, settles already-admitted commits, bounds/stops external and guest execution, and joins before releasing writer ownership.
- Reopen preserves pending, blocked, cancelled, and terminal evidence; close alone never synthesizes success or terminal receipt.

Proof: RPC waiter disconnects while work continues; abort during child effect cannot settle parent early; close during commit/guest/provider work permits safe reopen without duplicate work.
Test missing dependencies, queue preservation, quota/depth boundaries, caller abandonment, and writer handover through real processes/components.
All waits use explicit signals and deadlines; ordinary child work cannot detach.

### B5 · Reconnect an observer without losing committed transcript

Use revisioned bounded snapshot-plus-continuation in embedding and RPC; integrate existing inspection/observer readers.
Touch core committed-view/cursor projections, harness storage/inspection, RPC subscriptions, CLI inspection/observer; consume existing `smith-ui` vocabulary.

- Attach baseline and continuation coherently; positions include session and committed order, not volatile file offsets or progress counters.
- Page history and state with byte/item limits; retained chunks remain valid through eviction and maintenance.
- Backpressure may coalesce transient progress, never silently skip durable transcript; report recovery requirement and replay bounded missing history.
- Validate stale/cross-session/incompatible cursors and branch selection; coherent snapshots cannot mix file generations.
- Read-only observers can end their own waits but cannot submit/abort work; observation never schedules execution.
- This delivers local shared observation/RPC behavior; network daemon transports, pairing, and fleet replication remain outside scope.

Proof: built RPC observer stalls while writer advances, disconnects, reconnects at durable position, and reconstructs every committed transcript entry.
Public observer and `sessions get` agree after maintenance/branch changes; malformed cursor fails explicitly.
Property tests vary update batching/coalescing; allocation tests grow archived history with fixed visible/effective windows and separately account semantic state.

### B6 · Fork historical evidence without adopting source work

Expose independent writable fork through shared facade and existing session commands, including scoped-state policy.
Touch core session/work/state/cursors/store, harness inspection/runtime/storage, CLI fork/inspection, RPC receipt lookup.

- Preserve source-attributed visible intent/completion/unknown evidence; exclude later source completions from fork history.
- Do not copy live executor ownership or activate inherited work; new fork submissions receive fork-owned identities and key scope.
- Apply declared state policy at the fork point; core state is typed, extension payloads bounded and versioned, scratch absent.
- Keep source and fork write/commit boundaries independent; no cross-file transaction or parallel-running-branch assumption.
- Bound active state items/bytes and fork materialization; preserve archived evidence and queryable terminal receipts independently of caches.

Proof: fork during paused source effect, finish source, inspect both histories, submit new fork work, reopen both.
Fork retains original unknown evidence without adopting source execution, source completion stays source-only, new admission executes once under fork identity.
Test state copy/fresh policy, historical selection, unknown payload preservation, read-only source, and quota failure without partially published fork.

## Interface structure

Arrows are library dependencies; shared records keep harness/executor types out of core and RPC.
No new crate is required.

```mermaid
flowchart TB
  cli["smith-cli"] --> host["smith-harness: driver and effects"]
  cli --> rpc["smith-rpc: control and observation"]
  host --> core["smith-core: changes and projections"]
  rpc --> core
  core --> shared["smith: typed commands and receipts"]
  host --> shared
  rpc --> shared
```

## Verification and stop conditions

- Tests name owning spec; use real binary/files/sockets/components for public behavior, narrower properties for codecs, deduplication, bounds, and replay.
- Run fresh scoped `cargo x check`, `cargo x test`, `cargo x e2e`, `cargo x doc`, architecture gates after integrated slices; full relevant gates before completion.
- Snapshot receipts, diagnostics, replay and LF-only RPC; redact incidental IDs, not identity relationships or authority evidence.
- Deliberate format-version decision and reviewed golden regeneration accompany grouped semantics; no silent compatibility claims.
- Exact allocation/type bounds and scoped mutation tests target admission, commit validation, effect gating, ownership, cursor recovery, and fork isolation.
- Preserve instruction budgets; no unmeasured performance claim or automatic re-pin.
- Stop for missing component/loadout/request contracts, ambiguous state fork policy, unknown required semantics, or incompatible authorization; do not manufacture proof fixtures that bypass them.
- Stop if recovery needs current configuration, re-executed effects, whole-history residency, or caller acknowledgement as proof that valid bytes exist.
- Dependency/config/toolchain changes need separate approval; no implementation or plan approval is implied by this draft.