1 +#
Leaks Detector Reliability Roadmap 3 +> **For agentic workers:** Use /skill:writing-plans to create one detailed implementation plan per phase. Start with Phase 1 and proceed sequentially unless the user explicitly changes the order.
5 +**Goal:** Make `leaks` precision-first, durably hydratable across Pi lifecycle changes, and rigorously verified by deterministic, property, corpus, and real-agent-loop tests.
7 +**Design Spec:** [`docs/super/specs/2026-07-17-leaks-detector-reliability-design.md`](../specs/2026-07-17-leaks-detector-reliability-design.md)
9 +**Planning Strategy:** Detector semantics, durable lifecycle state, and final integration hardening have different failure modes and review surfaces. Three ordered phases keep every intermediate commit green while ensuring later persistence work builds on stable detector behavior.
13 +##
Phase 1: Detector Precision 15 +**Outcome:** Every detector uses precision-first semantics, every historical false positive has a deterministic regression, and every detector family has positive and negative fast-check properties runnable through a dedicated Mise task.
17 +**Why now:** Durable mappings must not preserve garbage produced by broad detectors. Detector semantics and placeholder candidates must stabilize before persistence is introduced.
21 +- Add `fast-check` and `@fast-check/vitest` as development dependencies.
22 +- Add `mise run fuzz` without adding it to the ordinary unit-test suite yet.
23 +- Replace JWT shape matching with Base64URL and JSON validation.
24 +- Parse URL, query, header, cookie, assignment, and private-key candidates conservatively.
25 +- Tighten environment and dotenv sensitive-name classification.
26 +- Add exact-span deterministic regressions and positive/negative properties for every detector family.
27 +- Update `extensions/leaks/SPEC.md` for precision-first detector behavior.
31 +- Durable encrypted mappings.
32 +- Placeholder format migration.
33 +- Branch, reload, resume, fork, or tree-navigation state.
34 +- Strict unresolved-placeholder blocking beyond existing behavior.
35 +- Final repository corpus and CI fuzz gate.
37 +**Key files/areas likely affected:**
39 +- `extensions/leaks/src/detectors.ts`: provider-specific token formats.
40 +- `extensions/leaks/src/structures.ts`: parsed structured candidates.
41 +- `extensions/leaks/src/dotenv.ts`: exact-value learning policy.
42 +- `extensions/leaks/src/safety.ts`: normalized sensitive-name classification.
43 +- `extensions/leaks/__tests__/`: deterministic incident regressions.
44 +- `extensions/leaks/__fuzz__/`: detector arbitraries and properties.
45 +- `extensions/leaks/SPEC.md`: normative detector contract.
46 +- `package.json`, `package-lock.json`, `mise.toml`: property-test dependencies and task.
50 +- Approved detector reliability design.
51 +- fast-check 4.9.0 and @fast-check/vitest 0.4.1.
55 +- Every built-in and structured detector has generated positive and negative cases with shrinking.
56 +- Historical dotted-code, source-code, key-substring, environment-metadata, and private-key-overrun incidents remain visible.
57 +- Exact provider tokens, parsed credentials, valid JWTs, and private keys still redact exact intended spans.
58 +- `mise run test`, `mise run fuzz`, `mise run lint`, and `mise run check` pass.
60 +**Phase boundary health:** The extension still uses its existing volatile placeholder lifecycle, but detector behavior is strictly safer and all existing user commands remain available. The repository remains expected-green.
64 +- Precision tightening can expose values previously redacted; the approved policy explicitly prefers this over false positives.
65 +- Test generators can reproduce implementation regexes and create circular evidence; generators must construct semantic values independently.
66 +- Generic assignment syntax overlaps source code; ambiguous values must remain visible.
68 +**Context notes:** Use TDD for every incident and detector family. Assert exact spans and round-trip behavior, not count lower bounds.
70 +##
Phase 2: Durable Lifecycle And Hydration 72 +**Outcome:** Every emitted V1 placeholder has an authenticated encrypted mapping on the active session branch, survives reload/resume/fork/tree navigation, and is the only kind of placeholder eligible for hydration.
74 +**Why now:** Stable detector semantics from Phase 1 define trustworthy mapping inputs. Persistence can now enforce the persist-before-redact invariant without preserving known false positives.
78 +- Introduce `leaks-generation-v1` and `leaks-mapping-v1` custom entries.
79 +- Introduce the exact V1 HMAC32 placeholder format.
80 +- Derive separate HMAC and AES-GCM keys from one generation root key.
81 +- Persist encrypted mappings before provider replacement.
82 +- Rebuild mappings from `SessionManager.getBranch()` on session start and tree navigation.
83 +- Define child-branch, clear-generation, fork, persisted-session, and in-memory-session behavior.
84 +- Hydrate only authenticated active-generation mappings.
85 +- Block every built-in or custom tool when an unresolved placeholder or incomplete input traversal remains.
86 +- Add TUI/RPC notifications and print/JSON stderr diagnostics.
87 +- Add deterministic and property-based lifecycle tests.
91 +- Full tracked-repository negative corpus.
92 +- Final two-minute fuzz tuning and CI inclusion.
93 +- Broad refactors outside `extensions/leaks`.
95 +**Key files/areas likely affected:**
97 +- `extensions/leaks/src/placeholders.ts`: generations, encryption, persistence records, restoration.
98 +- `extensions/leaks/src/engine.ts`: persist-before-redact and precision fallback.
99 +- `extensions/leaks/src/pi.ts`: session events, branch reconstruction, diagnostics, tool blocking.
100 +- `extensions/leaks/src/types.ts`: versioned records and diagnostic types.
101 +- `extensions/leaks/__tests__/`: lifecycle and Pi hook regressions.
102 +- `extensions/leaks/__fuzz__/`: model-based lifecycle properties.
103 +- `extensions/leaks/SPEC.md`: persistence and hydration contract.
107 +- Phase 1 detector semantics and property infrastructure.
108 +- Pi custom entries, `getBranch()`, `session_start`, and `session_tree` APIs.
109 +- Node crypto primitives only.
113 +- Every emitted placeholder has a durable authenticated mapping before provider use.
114 +- Reload, resume, fork, clear, tree navigation, and detector-change scenarios preserve only active-branch hydration.
115 +- Mapping collision, capacity, persistence, authentication, and decryption failures leave values unredacted and notify safely.
116 +- Unknown placeholders and incomplete traversals never reach tool execution.
117 +- In-memory sessions preserve reload behavior and explicitly report absent restart durability.
118 +- `mise run test`, `mise run fuzz`, `mise run lint`, and `mise run check` pass.
120 +**Phase boundary health:** The extension gains complete durable behavior without changing the external slash-command surface. Failures prefer provider visibility over unstable placeholders, so the session remains usable.
124 +- Append-only branch semantics can activate stale sibling mappings; reconstruction must use only `getBranch()`.
125 +- Co-located root keys and ciphertext provide obfuscation, not at-rest security; diagnostics and docs must not claim otherwise.
126 +- Tool traversal must fail closed without blocking ordinary validated JSON inputs.
127 +- Session-event ordering must rebuild state before provider or tool hooks run.
129 +**Context notes:** Keep versioned persistence logic isolated and small. Do not introduce a storage abstraction, crypto provider interface, or compatibility layer.
131 +##
Phase 3: Integration Hardening And CI Gate 133 +**Outcome:** Real Pi execution proves end-to-end hydration and lifecycle behavior, tracked production source guards against false positives, and the dedicated fuzz suite runs within two minutes as part of `mise run ci`.
135 +**Why now:** Final integration tests need the completed detector and persistence contracts. CI should gate only stable behavior, not transitional implementations.
139 +- Add real-agent-loop provider-to-tool hydration tests for built-in and custom tools.
140 +- Add process restart, resume, tree-navigation, branch, fork, corruption, and detector-change integration scenarios.
141 +- Add the tracked production-source negative corpus with safe masked failure output.
142 +- Strengthen positive fixture expectations from aggregate counts to exact labels and spans.
143 +- Complete model-based lifecycle properties and adversarial traversal coverage.
144 +- Tune fuzz execution to the two-minute budget with reproducible seeds and replay paths.
145 +- Add `mise run fuzz` to `mise run ci` while keeping it outside `mise run test`.
146 +- Update README and changelog behavior documentation.
150 +- New detector settings.
151 +- External parser or crypto dependencies.
152 +- At-rest session encryption.
153 +- Unrelated extension modernization.
155 +**Key files/areas likely affected:**
157 +- `extensions/leaks/__tests__/harness.test.ts`: real agent-loop execution.
158 +- `extensions/leaks/__tests__/`: restart, branch, corpus, and fixture assertions.
159 +- `extensions/leaks/__fuzz__/`: final property and model suites.
160 +- `test/harness.ts`: only if a missing real Pi lifecycle capability cannot be exercised otherwise.
161 +- `mise.toml`: CI fuzz gate.
162 +- `extensions/leaks/README.md`, `extensions/leaks/CHANGELOG.md`: user-facing behavior.
166 +- Phase 1 precision-first detectors.
167 +- Phase 2 durable mappings and strict hydration.
171 +- Scripted model placeholders become original bytes before actual tool execution.
172 +- No unexpected detector matches occur in tracked production source.
173 +- Corrupt or unknown state produces safe diagnostics and no unstable placeholder execution.
174 +- Fuzz failures report replayable seed and path data.
175 +- `mise run fuzz` completes within two minutes.
176 +- `mise run ci` passes.
178 +**Phase boundary health:** This is the release boundary. No later phase is required for the approved design to function safely.
182 +- Harness shortcuts can accidentally test extension emitters instead of the real agent loop; assertions must inspect actual tool implementation inputs.
183 +- A repository corpus can become an allowlist graveyard; intentional positives belong in explicit fixtures, not broad exclusions.
184 +- Time-limited fuzzing can become flaky; deterministic minimum coverage and reproducible failure metadata remain mandatory.
186 +**Context notes:** Treat any integration failure as a product bug, not a test inconvenience. Do not weaken generators, corpus scope, or exact assertions to make CI green.