--- id: SMH-PLAN-KFFL0_FT type: plan title: "Paged Session Storage, Observer Mode, and Inspection CLI" spec: SMH-SPEC-SPEC0001 status: approved --- # Paged Session Storage, Observer Mode, and Inspection CLI ## Status Bounded storage, observer mode, inspection commands, and unique default session identities are implemented in the current tree. The original gate and exploratory evidence remain below as historical record. The native `portable-pty` observer harness was tested on Linux. Native Win32 and macOS CI remain explicitly deferred because Luci runs Linux containers only. Live compaction releases superseded context from resident agent state while durable readers retain full replay history and fork cleanup leaves no source-session mutation. `cargo x check` passed 324 tests and `cargo x e2e` passed 25 tests. Matching-container Callgrind accepted six typed-codec, agent, and fork reductions, retained the unchanged `from_frames` pin within its band, and replaced the RPC unit with `ping_batch_128`. Storage-worker `eval --mock` startup and CLI/dependency `--help` startup increases are deliberately accepted; residual `--help` attribution remains incomplete. ## Intent Keep session access proportional to the operation, not total history size. Support independent Smith processes, read-only observers, and Unix-friendly inspection. Replace the borrowed-CBOR-only proposal with a storage redesign. ## Decisions - One self-contained append-only binary file per session; no SQLite or shared catalog/index. - Unique session identity and filename; default runs create new sessions, resume selects existing ones explicitly. - One exclusive writer per session; lock contention opens an observer instead of waiting indefinitely. - Explicit bounded reads on storage workers; UI consumes prepared immutable data, never performs storage I/O. - No mmap initially; retain a storage boundary allowing later measured evaluation. - No Rust memory dumps: explicit widths, byte order, tags, lengths, offsets, and validation define the format. - Chosen encoding: custom fixed metadata and typed content, offset-addressed raw text/output, canonical CBOR only for arbitrary structured payloads. Borrowed access, bounded reads, and complete semantic roundtrips are acceptance requirements. Measured codec CPU savings do not establish an end-to-end session speedup. - Derived indexes and checkpoints live inside the session file, not separately synchronized files. - Serde stays; no simd-json adoption or blanket derive removal. Prototype permits format replacement without migration support. A deliberate frame-version bump accompanies new golden fixtures; deterministic encoding and recovery laws remain mandatory. ## Historical implementation gate At approval time, the Smith Specification covered the chosen encoding, bounded working sets, observer access, and inspection CLI. Checkpoint addressing, durable publication, branch indexing, and observer/maintenance coordination were the original pre-implementation gate items. Approval covered the chosen direction and execution order; those gate statements are retained as historical context. Dependency-file changes still require separate approval. ## Order 1. Pin existing session identity, branching, replay, tool recording, corruption recovery, and secret-redaction behavior. Profile open/resume, visible transcript access, context construction, replay, and cross-session inspection. Record memory/allocation and instruction baselines; distinguish file reads from decoded retention. 2. Design the binary layout and append-publication protocol; no further codec-choice experiment is prerequisite. Specify checksums, complete-record visibility, durability acknowledgements, parent references, offset validation, and selected-branch recovery. Define checkpoint discovery, cadence, and fallback scanning; incomplete or invalid checkpoints cannot hide valid history. Reject reverse magic-marker scanning as authoritative discovery: valid checkpoint-shaped payload bytes can spoof it. Separate bounded memory from reopen scan time; do not claim bounded tail recovery without proving checkpoint discovery. Distinguish fast opening from full integrity verification; validate records on access and provide explicit full-file verification semantics. 3. Implement bounded record access and embedded lookup/checkpoint support. Load requested records or batches, not every payload or a mandatory whole-history index. Keep derived indexes recoverable from authoritative records. Bound decoded caches and retained chunks; define eviction and view lifetimes. 4. Integrate independent working sets: visible UI history, effective agent context, and sequential replay/export. Traverse selected branches and compaction boundaries without decoding unrelated branches. Remove eager request-history materialization where needed; paging does not eliminate the effective-context memory floor. 5. Implement exclusive writer acquisition and read-only observer opening. Observers read only published complete records; an incomplete live tail is not treated as permanent damage. Define safe repair/replacement coordination before allowing observers alongside maintenance. 6. Add observer UI behavior and session inspection commands through the same bounded storage API. Implement unique default session creation instead of fixed eval/RPC filenames. 7. Run binary/frontend acceptance tests and full gates. Review measured memory and instruction changes; deliberately re-pin budgets in the CI image. ## Observer UI Show persistent read-only state and explain that another instance owns writing. Permit transcript navigation, search, export, and explicit fork into a new writable session. Disable prompt submission, tool execution, and mutations of the observed session. Local navigation never changes the writer's persisted selection. Follow-live is an explicit toggle; updates consume only complete published records. Writer exit never silently promotes an observer to writer. ## Session inspection CLI - `smith sessions list`: enumerate session identities and summary metadata without decoding complete histories. - `smith sessions grep `: search recorded content across sessions incrementally; include session and entry identity in results. - `smith sessions get `: inspect one session through bounded sequential reads. Human-readable output is the default; `--json` emits LF-delimited JSON records for scripts and agents. Keep stdout data-only; diagnostics go to stderr; no interactive dependency or terminal escape sequences in piped output. Inspection never repairs, truncates, or acquires writer ownership. Reuse observer snapshot boundaries for active sessions; continuous following is explicit, never default. Define search syntax, branch scope, ordering, exit codes, broken-pipe handling, and corrupt-session reporting before implementation. Start with minimal options; add filters and projections only for concrete consumers. Do not bypass existing secret masking or expose credentials through inspection. ## Coordination The JSON-boundary plan owns semantic payload types and shared borrowed structured-value access. This plan owns file layout, storage access, publication, observer behavior, and inspection integration. Keep CBOR and filesystem implementation types out of shared domain APIs. ## Verification - Independent sessions remain writable concurrently; competing writers for one session yield one writer and read-only observers. - Observer navigation and inspection cannot mutate source history; explicit fork creates independent writable history. - Real multi-process tests cover append visibility, interrupted records, writer exit, and maintenance coordination using signals and deadlines, never sleeps or polling waits. - Malformed lengths, offsets, tags, parent references, checksums, and nested payloads fail within bounds. - Crash-point tests cover append publication, checkpoint publication, reopen, repair, and fork creation. - Roundtrip/canonicalization properties and versioned golden fixtures prove deterministic storage and replay. - Large-history tests bound memory for listing, visible-range access, grep, and export independently of total payload size. - Context-building tests measure its distinct working set rather than claiming UI-sized memory for agent execution. - Built-binary tests exercise list, grep, get, human output, JSONL pipelines, damaged sessions, and active-session inspection. - Frontend harness proves observer status, disabled mutations, navigation, follow-live, and explicit fork behavior. ## Experiment evidence Exploratory results only; probes, fixtures, builds, and raw outputs deleted at human request. No retained executable regression gate; implementation must establish permanent coverage. ### Storage access Selective reads held 64 KiB peak tracked heap across synthetic histories up to approximately 200 MB, with both raw and CBOR payloads. Sequential scans stayed bounded; owned CBOR decoding still caused transient allocations. Multi-process checks supported exclusive writers and read-only observation of complete appends. 356 truncation boundaries passed, but checkpoint-shaped bytes inside valid payloads fooled reverse discovery. An 8 MiB uncheckpointed tail required approximately 16 MiB of requested reads. Later checkpoints bypassed older corruption; fast opening is not full integrity verification. ### Codec comparison Used actual Smith EntryContent and EntryFrame content encoding across all four variants and all message block variants. Compared current owned decoding, borrowed current CBOR, custom typed binary content, and compact CBOR arrays with opaque nested CBOR payloads. Roundtrips preserved complete typed content, including empty values, integer extremes, UTF-8, and nested arbitrary JSON. Selective-read measurements inspected visible text/name fields on prevalidated in-memory records, not filesystem I/O or full provider context. For 1,024 visits to 256 mixed entries with 4 KiB text, Callgrind counted approximately 30.65 million instructions for current owned decoding, 13.20 million for borrowed current CBOR, 4.77 million for compact borrowed CBOR, and 2.75 million for custom borrowed binary. All three borrowed readers allocated zero times; current owned decoding allocated 15,104 times. Borrowed instruction counts repeated exactly; owned counts varied slightly. Custom binary therefore reduced measured instructions even against compact borrowed CBOR, not only against owned decoding. Compact CBOR was slightly smaller than custom binary for this corpus; custom encoding has no established space advantage. Environment: desktop Ryzen 5 5600X, Linux 7.2.7-1-cachyos, rustc/Cargo 1.98.1, release profile, ciborium 0.2.2, Valgrind 3.25.1, smith-alloc accounting. Warm timings used nine interleaved rounds and repeated medium/out-of-cache cases; no wall-clock numbers retained. Small/medium selective reads favored custom encoding; large-payload timing ranges overlapped. Current CBOR borrowed lookup used generic map-field searches; further CBOR specialization remains an alternative. ### Remaining proof Custom typed encoding is selected on measured local CPU evidence, not a proven end-to-end storage speedup. Arbitrary structured payloads retain CBOR; removing CBOR entirely is not the goal. The current implementation covers bounded reads, durable publication, observer access, inspection commands, and CLI session identity through focused tests and binary e2e tests. No mmap or power-loss proof exists, and repair/observer coordination remains limited to the implemented append and live-tail boundaries. The native portable-PTY proof is Linux-tested; native Win32 and macOS CI are deferred because Luci provides Linux containers only. Matching-container instruction budgets were deliberately re-pinned for the changed batch RPC unit, six typed-codec/agent/fork reductions, and accepted startup increases. The `--help` residual remains an incomplete attribution, not a claimed optimization result. ## Exit criteria and current state Custom typed records and CBOR arbitrary payloads preserve complete semantic roundtrips. Single-session access borrows required content without intermediate trees; sequential operations use bounded batches. Session files are self-contained, recoverable, and independently writable. Observers are useful without ambiguous write ownership. Humans and agents inspect sessions through composable CLI output. No shared index synchronization, query engine, or mandatory mmap. These criteria describe the implemented scope; this plan does not assert that every repository gate or perf budget is complete.