4 +**Status:** Draft for review
5 +**Objective:** Spec a Pi-idiomatic extension named `cmprss` from Headroom's useful ideas, without external servers, host wrappers, legacy compatibility, or non-Pi baggage.
9 +Researched repository: `https://github.com/headroomlabs-ai/headroom`.
10 +Local snapshot: `/tmp/pi-research/headroom`.
12 +Commit: `660fa8cfb68f02f6fb71aa3a6b9d646420956a0e`.
14 +Headroom files inspected:
16 +- `README.md` for product scope: library, proxy, wrap, MCP, CCR, dashboard, memory, learn, output shaping.
17 +- `pyproject.toml` for Python dependencies and optional extras.
18 +- `Cargo.toml` for Rust workspace split: core, proxy, Python bridge, parity.
19 +- `sdk/typescript/README.md`, `src/index.ts`, `src/compress.ts`, and `src/client.ts` for TypeScript SDK behavior.
20 +- `wiki/typescript-sdk.md` for the explicit statement that TypeScript compression calls the proxy.
21 +- `wiki/proxy.md` for server mode, run modes, endpoint shape, and legacy mode aliases.
22 +- `wiki/ccr.md` for Compress-Cache-Retrieve.
23 +- `wiki/compression.md` for content routing, JSON/code/log/text structure preservation.
24 +- `headroom/compress.py` for the direct Python compression API.
25 +- `headroom/pipeline.py` for pipeline stages.
26 +- `headroom/transforms/content_router.py` for routing concepts.
27 +- `headroom/transforms/smart_crusher.py`, `search_compressor.py`, `log_compressor.py`, and `code_compressor.py` for concrete compression families.
29 +Pi and pi-ext files inspected:
31 +- `/home/me/Workspace/dotfiles/pi-ext/AGENTS.md` for mise-only tasks, compact tool widget rules, cross-platform rules, and markdown style.
32 +- `/home/me/.pi/agent/skills/pi/references/extensions.md` for extension hooks, commands, tools, UI, and provider APIs.
33 +- `/home/me/.pi/agent/skills/pi/references/extension-lifecycle.md` for reload/session safety.
34 +- `/home/me/.pi/agent/skills/pi/references/patterns.md` for custom compaction and hook examples.
35 +- `mise.toml` for required tasks: `mise run fmt`, `lint`, `test`, `check`, `ci`, `readme`, `changelog`.
36 +- root `package.json` for workspace and Pi package exports.
37 +- `extensions/ctx/index.ts`, `README.md`, `package.json`, and `__tests__/ctx.test.ts` for minimal `tool_result` compression.
38 +- `extensions/terse/index.ts`, `README.md`, and `package.json` for deterministic prompt injection.
39 +- `extensions/tldr/index.ts` and tests for bounded UI/widget rendering.
40 +- `extensions/goldfish/index.ts`, `README.md`, and `package.json` for session entries, commands, state replay, and UI state.
41 +- `extensions/builtin-tools/index.ts` for small command/hook style.
42 +- `scripts/extensions.mjs` for extension package checks and README generation expectations.
46 +Headroom is a broad context-compression system.
47 +Its strongest ideas are useful for Pi: content-aware compression, reversible retrieval, and visible savings.
48 +Its implementation is not Pi-idiomatic for this repo.
50 +The TypeScript SDK is only an HTTP client.
51 +It calls `POST /v1/compress` on a local Headroom proxy and does not run compression in Node.
52 +The full local implementation lives across Python, Rust, maturin, native extensions, FastAPI, provider shims, optional ML stacks, telemetry, dashboards, memory systems, agent wrappers, and MCP.
53 +That is the wrong shape for `pi-ext`.
55 +`cmprss` should be a Pi extension that hooks Pi itself.
56 +It should compress oversized Pi context before it reaches the model, expose a Pi retrieval tool when exact omitted bytes are needed, and report small bounded savings.
57 +It must not start or require an external server.
61 +- Build `cmprss` as a Pi-only TypeScript extension under `extensions/cmprss`.
62 +- Run entirely inside the Pi extension runtime.
63 +- Compress large tool results before they enter model context.
64 +- Preserve enough structure for the model to decide whether to continue or retrieve exact content.
65 +- Provide reversible retrieval for compressed blocks through a Pi tool.
66 +- Keep all state session-local and reload-safe.
67 +- Use deterministic Node/TypeScript code only.
68 +- Allow tiny npm parsing dependencies when they materially improve compression correctness.
69 +- Follow existing extension shape: package, README, tests, small files, no ad-hoc scripts.
70 +- Keep collapsed UI/tool output compact; put verbose lists behind expanded tool rendering if custom rendering is added.
71 +- Stay cross-platform: Node APIs only, no shell helpers.
78 +- No Rust or native extension.
79 +- No `headroom-ai` package dependency.
81 +- No large runtime dependency stack.
82 +- No dependency that starts subprocesses, downloads models, opens sockets, or needs native compilation.
83 +- No provider wrapper.
84 +- No `wrap` or `unwrap` commands for Claude, Codex, Cursor, OpenCode, or any other host.
86 +- No telemetry, pricing, subscription, quota, OTEL, Langfuse, or data flywheel.
87 +- No cross-agent memory.
88 +- No `learn` feature.
89 +- No ML compression, embeddings, ONNX, PyTorch, HuggingFace, Magika, tree-sitter, ast-grep, or vector DB in v1.
90 +- No legacy Headroom env vars or mode aliases.
91 +- No compatibility with old Headroom config files.
92 +- No mutation of user files.
93 +- No compression of data needed for exact edits unless explicitly recoverable and safe.
95 +##
Headroom concepts to keep 97 +###
Keep: content routing 99 +Headroom routes content to different compressors.
100 +`cmprss` should keep this idea, but use small deterministic detectors:
102 +- JSON object/array.
103 +- Search result lines.
104 +- Logs/test/build output.
112 +###
Keep: structure preservation 114 +The model needs shape, not full bulk.
115 +`cmprss` should preserve keys, filenames, line numbers, error lines, summaries, and head/tail anchors.
117 +###
Keep: CCR, but Pi-native 119 +Headroom's CCR idea is good: compressed output includes a retrieval marker and the original remains available.
120 +`cmprss` should implement a small Pi-native version:
122 +- Store the original compressed block under a hash.
123 +- Emit a marker with hash, original size, compressed size, and omitted count.
124 +- Register a `cmprss_retrieve` tool that returns the exact original block by hash.
126 +No injected provider tools.
127 +No response-handler loop.
128 +No automatic second provider call.
129 +Pi already has tools; the agent can call the retrieval tool normally.
131 +###
Keep: savings metrics 133 +Report approximate bytes and token estimates.
134 +Use `chars / 4` or UTF-8 bytes / 4.
135 +Do not pretend exact tokenizer parity.
137 +##
Headroom concepts to reject 139 +###
Reject: proxy mode 141 +Headroom's proxy is the main production path.
142 +It intercepts provider requests and exposes OpenAI/Anthropic-compatible endpoints.
143 +`cmprss` must not do this.
144 +Pi already owns the agent loop and tool lifecycle.
145 +Hooking `tool_result` and optionally `context` is simpler and more direct.
147 +###
Reject: TypeScript SDK dependency 149 +The SDK calls the proxy.
150 +Using it would recreate the external server requirement.
152 +###
Reject: provider adapters 154 +Pi providers already exist in Pi.
155 +`cmprss` should not register or proxy models.
157 +###
Reject: native and ML stack 159 +Headroom's high-end compression depends on Python/Rust/native/ML pieces.
160 +This repo favors small Pi extensions.
161 +The useful first version is deterministic and local.
163 +###
Allow: tiny parser dependencies 165 +Tiny npm dependencies are allowed when they reduce brittle parsing code.
166 +They must be pure JavaScript or already-normal Node packages, cross-platform, deterministic, and small.
167 +Good candidates: `diff` for diff parsing and hunk handling, and `jsonc-parser` for tolerant JSON/JSONC parsing.
168 +Possible later candidate: `@babel/parser` for opt-in JavaScript/TypeScript structural summaries.
169 +Reject any dependency that brings ML models, native build chains, network fetches, or broad framework surface.
171 +###
Reject: legacy compatibility 173 +No accepted aliases like `token_headroom` or `cost_savings`.
174 +No old Headroom env vars.
176 +No hidden compatibility shims.
177 +If a setting is wrong, fail clearly or ignore with a bounded warning.
181 +###
Default behavior 183 +When enabled, `cmprss` watches Pi `tool_result` events.
184 +If a result is below the configured byte threshold, it passes through unchanged.
185 +If a result is above threshold and the tool/result class is compressible, it replaces the result content with compressed text and adds structured details.
187 +Default mode: enabled.
188 +Default threshold: `20000` UTF-8 bytes.
189 +Default persisted retrieval store capacity: `64` blocks per session branch.
190 +Default maximum retrieved bytes: unlimited for exact hash retrieval, because retrieval is explicit.
192 +###
Compressible tools 194 +Compress by default:
200 +- custom tools with text results, if their result text exceeds threshold and they are not marked unsafe
202 +Conservative handling:
204 +- `read` is not compressed when the input has `offset`, `limit`, or `l`.
205 +- `read` full-file output may be compressed only with a marker that instructs the agent to use `read offset/limit` or `cmprss_retrieve` before editing hidden content.
206 +- `edit`, `write`, and mutation result metadata are never compressed.
208 +Rationale: Pi editing depends on exact `oldText` matching.
209 +Do not hide bytes the model may need for an edit unless exact retrieval is obvious.
215 +Supported subcommands:
228 +- Bare `/cmprss` prints status.
229 +- `status` prints enabled state, threshold, stored block count, and session savings.
230 +- `on` enables compression for the current session branch.
231 +- `off` disables compression for the current session branch.
232 +- `stats` prints compressed block summaries with bounded rows.
233 +- `clear` drops stored originals for the current session branch and resets counters.
235 +Do not add `wrap`, `proxy`, `server`, `dashboard`, `learn`, `doctor`, or `perf`.
239 +Register `cmprss_retrieve`.
251 +- Exact original text for the hash.
252 +- Metadata: hash, original bytes, compressed bytes, compressor, source tool, created time.
254 +If the hash is unknown, throw or return an error result clearly saying `cmprss: unknown hash`.
255 +Do not fake success.
259 +Every compressed block includes a visible marker:
262 +[cmprss: <tool> <originalBytes>B → <compressedBytes>B, saved ~<tokens>t, hash=<hash>]
263 +Use cmprss_retrieve(hash="<hash>") for exact original text.
269 +Before editing hidden content, retrieve exact text or use read offset/limit.
274 +Pi settings namespace:
280 + "thresholdBytes":
20000,
281 + "maxStoredBlocks":
64,
283 + "tools": [
"bash",
"grep",
"find",
"ls",
"read",
"custom"]
290 +- Settings are read from Pi settings through the extension API if available.
291 +- Session commands override settings for the current branch only.
292 +- Invalid values fall back to defaults and may notify once in UI sessions.
293 +- No environment variables in v1.
294 +- No legacy aliases.
298 +Use custom session entries for branch-aware state.
300 +Entry type: `cmprss-state`.
305 +type CmprssStateEvent = 306 + | { op
: "enabled"; enabled
: boolean; at
: number }
307 + | { op
: "clear"; at
: number }
308 + | { op
: "block"; meta
: CmprssBlockMeta; originalText
: string; at
: number };
311 +The original text is persisted in the custom entry for every compressed block.
312 +For Pi bash results that arrive pre-truncated with `details.fullOutputPath`, persist that file content instead of the visible truncated text.
313 +For v1, persist exact originals in Pi session custom entries so retrieval survives Pi restarts.
314 +On reload/resume, replay metadata and originals from the current branch before accepting `cmprss_retrieve` calls.
315 +When a hash is absent from the replayed branch, `cmprss_retrieve` must fail loud with `cmprss: unknown hash`.
317 +This is intentionally simpler than Headroom's SQLite/Redis memory stack.
318 +It bloats the session file by design, because restart-stable retrieval is required for good results.
320 +Human decision: v1 must persist originals in session history.
321 +No in-memory-only retrieval mode.
325 +When `cmprss.status` is true and `ctx.hasUI` is true, set a compact status segment:
328 +cmprss: <savedTokens>t
337 +Do not add a full dashboard.
338 +Do not add noisy notifications for every compression.
339 +Notify only on first compression in a session and on command changes.
341 +###
Details metadata 343 +When a tool result is compressed, merge existing details and add:
346 +type CmprssDetails = {
349 + compressor
: "json" | "search" | "log" | "text";
350 + originalBytes
: number;
351 + compressedBytes
: number;
352 + tokensSavedEstimate
: number;
353 + sourceTool
: string;
358 +Do not drop existing details.
362 +Port Headroom compressor ideas in this order:
364 +1. Search compressor.
365 + High ROI, easy TypeScript, preserves paths and line numbers, no dependency needed.
366 +2. Log/test compressor.
367 + Huge savings on failing builds and CI output, deterministic, no dependency needed.
368 +3. JSON/table crusher.
369 + Worth it for tool payloads and API dumps; use `jsonc-parser` only if plain parsing is too brittle.
371 + Important for pi-ext review workflows; use `diff` if it shrinks hunk parsing and preserves headers reliably.
372 +5. Stacktrace compressor.
373 + Implement as a specialized log path that keeps full causal exception chains.
377 +- ML/Kompress/ONNX: too heavy, opaque, slow, and not Pi-extension-shaped.
378 +- Code AST compressor: risky for edit workflows; only consider opt-in later.
379 +- CacheAligner: provider-cache concern, less useful for Pi tool-result compression.
380 +- Memory/vector/TOIN: sprawling and out of scope.
382 +##
Compression algorithms 388 +- Text parses as JSON object or array.
389 +- Or text contains a JSON-looking top-level array/object after a known tool envelope.
393 +- For arrays: keep first N, last N, and anomalous/error-looking items.
394 +- For objects: preserve keys and short scalar values; summarize long arrays/strings.
395 +- Preserve valid JSON when possible.
396 +- If preserving valid JSON would cost too much code, emit a clear text summary plus JSON snippets.
400 +- Keep first 5 array items.
401 +- Keep last 5 array items.
402 +- Keep up to 10 error/anomaly items.
403 +- Cap rendered JSON at threshold target.
407 +- Keys or values containing `error`, `fail`, `fatal`, `warn`, `exception`, `denied`, `timeout`, `missing`, `deprecated`, `critical`.
408 +- Objects with unique key sets compared to neighbors.
410 +###
Search compressor 415 +path/to/file.ts:42:match text
416 +path/to/file.ts-43-context text
422 +- Keep first and last match per file.
423 +- Keep matches with error/anomaly hints.
424 +- Cap files and matches.
425 +- Preserve file paths and line numbers exactly.
430 +- 5 matches per file.
438 +- Contains build/test/log markers such as `ERROR`, `FAIL`, stack traces, assertion failures, `npm ERR`, `pytest`, `cargo`, `warning`, `Traceback`, `Exception`.
442 +- Keep command exit summary when present.
443 +- Keep first 20 lines.
444 +- Keep all high-signal error/failure lines up to cap.
445 +- Keep warning lines up to cap.
446 +- Keep traceback chunks with surrounding context.
447 +- Keep last 40 lines.
448 +- Dedupe repeated low-signal lines.
461 +- Unified diff text.
462 +- Git diff output from `bash` or review tools.
466 +- Preserve file headers exactly.
467 +- Preserve hunk headers exactly.
468 +- Keep changed lines.
469 +- Keep conflict markers and lines containing error/anomaly hints.
470 +- Drop large unchanged context blocks first.
471 +- Emit omitted context counts per file or hunk.
475 +- Use `diff` if it makes parsing smaller and more reliable than handwritten hunk parsing.
477 +###
Stacktrace compressor 481 +- Python tracebacks.
482 +- JavaScript/TypeScript stack traces.
483 +- Java/Kotlin stack traces.
484 +- Rust panic traces.
489 +- Keep exception type and message.
490 +- Keep first and last frame.
491 +- Keep application frames over dependency frames.
492 +- Keep chained causes: `caused by`, `During handling`, `The above exception`, `Suppressed`.
493 +- Compress repeated dependency frames with counts.
501 +- Keep head and tail.
502 +- Include omitted line/byte counts.
503 +- Preserve code fences if they begin or end in kept windows.
504 +- Never invent summary text beyond counts and marker.
520 +│ ├── compressors.ts
526 + ├── compressors.test.ts
528 + ├── harness.test.ts
536 +##
Dependency policy 538 +Allowed npm deps must be small, deterministic, cross-platform, and compression-specific.
539 +Prefer no dep when plain TypeScript is clearer.
540 +Use a dep only when it improves correctness enough to shrink code.
542 +Approved initial candidates:
544 +- `diff` for unified diff parsing and hunk selection.
545 +- `jsonc-parser` for tolerant JSON/JSONC parsing if plain `JSON.parse` misses common tool output.
549 +- `@babel/parser` for opt-in JS/TS AST summaries after v1 proves value.
553 +- ML/model runtimes.
554 +- Native build requirements.
555 +- Network clients for compression.
557 +- Headroom SDK/client packages.
561 +`extensions/cmprss/package.json`:
563 +- `name`: `@bugabinga/pi-ext-cmprss`
565 +- `main`: `index.ts`
567 +- peer dependencies only for Pi packages and `typebox` if the retrieve tool uses TypeBox schemas
568 +- dev dependencies: `vitest`, `@marcfargas/pi-test-harness`
570 +Root `package.json` already exports `extensions/*/index.ts`, so no root package change is required for extension discovery unless a skill is later added.
571 +No skills are needed in v1.
573 +##
Hook architecture 577 +- `session_start` to replay session state and set status.
578 +- `session_shutdown` to clear process-local originals for the old runtime.
579 +- `tool_result` to compress eligible text results.
580 +- `cmprss_retrieve` tool.
581 +- `/cmprss` command.
583 +Do not capture stale `ctx`.
584 +Keep process globals reload-safe or clear them on `session_shutdown`.
588 +- Compression failure returns the original result unchanged and records a bounded warning in details.
589 +- Retrieval miss fails loud.
590 +- Invalid command arguments print help.
591 +- Invalid settings fall back to defaults.
592 +- Aborted tool calls respect `signal.aborted` where available.
593 +- Hash collisions are practically ignored with SHA-256 truncation, but store compares full hash internally.
595 +No silent lossy fallback for retrieval.
596 +No background workers.
599 +##
Security and secret handling 601 +`cmprss` stores exact tool output in process memory.
602 +That may include secrets already exposed to the model by a tool result.
603 +It must not print stored originals in stats, UI, or details.
604 +Only `cmprss_retrieve(hash)` returns the exact original.
606 +Do not scan files independently in v1.
607 +Only handle Pi tool outputs already produced.
611 +Use `mise run test -- cmprss` or the repo's normal test task.
612 +Do not bypass mise for listed repo tasks.
616 +- Harness registers one command, one tool, and expected hooks.
617 +- Small results pass through unchanged.
618 +- Disabled mode passes through unchanged.
619 +- Oversized text compresses and stores retrievable original.
620 +- Retrieval returns exact original for a known hash.
621 +- Retrieval miss fails loud.
622 +- Existing details are preserved.
623 +- Explicit `read` windows are not compressed.
624 +- Mutation tools are not compressed.
625 +- JSON array keeps anchors and emits counts.
626 +- Search output preserves paths and line numbers.
627 +- Log output preserves error/traceback/failure lines.
628 +- Session replay applies latest enable/disable entry.
629 +- Clear tombstones prior persisted originals and resets counters.
630 +- Restart/replay restores `cmprss_retrieve` for prior compressed blocks.
632 +Required checks before implementation is considered done:
647 +##
README requirements 649 +`extensions/cmprss/README.md` should include:
651 +- Load through root `pi-ext` package.
652 +- Commands/tools/settings/hooks list.
653 +- Compression safety rules.
654 +- Retrieval example.
655 +- Statement that no server, proxy, Python, Rust, ML, MCP, telemetry, or dashboard exists.
657 +##
Acceptance checks 659 +Observable acceptance for the spec:
661 +- The design names the exact Headroom concepts retained.
662 +- The design names the exact Headroom concepts rejected.
663 +- The design avoids external server/proxy architecture.
664 +- The design uses Pi hooks/tools/commands/session state directly.
665 +- The design identifies existing pi-ext conventions inspected.
666 +- The design includes settings, command surface, retrieval tool, state model, UI, tests, and release checks.
667 +- The design records the user decision that originals must persist across Pi restarts.
669 +Observable acceptance for future implementation:
671 +- Loading the extension does not start a server or spawn a process.
672 +- `cmprss_retrieve` works without network access.
673 +- A large `bash` output is compressed before the next model turn.
674 +- The original large output can be retrieved by hash after restarting Pi.
675 +- Explicit `read offset/limit` output is never compressed.
676 +- `mise run ci` passes.
680 +V1 persists exact originals in Pi session history.
681 +This makes compressed blocks restart-stable and branch-local.
682 +The accepted cost is larger session files.