repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

feat: add cmprss extension

f33dd9d467902569b4a975bbfd7eaa9bfe7d293b

Oliver Jan Krylow <oliver@bugabinga.net> ·

unsigned

Browse filesCompare with HEAD

Changed files (10)

CHANGELOG.md

Modified
2 2 # Changelog
4 4 ## Unreleased
6 +- Add a `cmprss` Pi extension design spec adapted from Headroom concepts, keeping deterministic content-aware tool-result compression, Pi-native retrieval markers, session-persisted originals for restart-stable retrieval, compact savings UI, commands, settings, tests, and acceptance checks while rejecting proxy/server, SDK dependency, Python/Rust/ML/MCP, telemetry, dashboards, provider wrappers, and legacy compatibility
7 +- Implement `cmprss` tool-result compression with compact retrieval markers, pre-truncated Pi bash `details.fullOutputPath` persistence, and restart-stable `cmprss_retrieve` access to exact original text
8 +- Validate `cmprss` retrieval across runtime reloads for compressed large bash outputs
9 +- Validate `cmprss` disabled output passthrough and unknown-hash retrieval misses
10 +- Fix `cmprss` stacktrace compression so stack frames are classified before search-like file/line matches, preserving stack causes and retrieval exactness
11 +- Add a `lazy` Pi extension design spec, adapted from Ponytail 4.8.4 but Pi-only for this repo, covering lazy modes, settings, commands, skills, session state, prompt injection, post-turn `lazy:` marker notifications, performance constraints, tests, and same-or-better parity while dropping legacy Ponytail command names, config migration, environment overrides, and host portability shims
12 +- Implement the `lazy` Pi extension with `/lazy` modes, standalone deactivation phrases, prompt injection, session-local mode replay, active/inactive status UI, deterministic help/gain/debt commands, review/audit skill handoffs, post-agent `lazy:` marker notifications, README, skills, root skill export, git probe timeouts, and unit/harness coverage
13 +- Validate `lazy` on a larger todo-CLI A/B smoke: same passing behavior with 103 LOC versus 154 baseline LOC, zero deps in both variants
14 +- Add global Pi prompts adapted from Matt Pocock's productivity skills: `gr`,
15 + `handoff`, and `sensei`; add the global `grilling` skill plus supporting
16 + teaching reference files, merge skill-writing guidance into the Pi skill, and
17 + include MIT license copies
6 18 - Add ultra `subagentExtensions` setting (default `"all"`): workflow sub-agents
7 19 load the host's extensions so a step can grant extension-provided tools (e.g.
8 20 `web_search`), while the step `tools` allowlist stays the only gate on what is

docs/superpowers/specs/2026-07-03-cmprss-pi-extension-design.md

Added
new file mode 100644
1 +# cmprss Pi extension
3 +**Date:** 2026-07-03
4 +**Status:** Draft for review
5 +**Objective:** Spec a Pi-idiomatic extension named `cmprss` from Headroom's useful ideas, without external servers, host wrappers, legacy compatibility, or non-Pi baggage.
7 +## Source evidence
9 +Researched repository: `https://github.com/headroomlabs-ai/headroom`.
10 +Local snapshot: `/tmp/pi-research/headroom`.
11 +Branch: `main`.
12 +Commit: `660fa8cfb68f02f6fb71aa3a6b9d646420956a0e`.
14 +Headroom files inspected:
16 +- `README.md` for product scope: library, proxy, wrap, MCP, CCR, dashboard, memory, learn, output shaping.
17 +- `pyproject.toml` for Python dependencies and optional extras.
18 +- `Cargo.toml` for Rust workspace split: core, proxy, Python bridge, parity.
19 +- `sdk/typescript/README.md`, `src/index.ts`, `src/compress.ts`, and `src/client.ts` for TypeScript SDK behavior.
20 +- `wiki/typescript-sdk.md` for the explicit statement that TypeScript compression calls the proxy.
21 +- `wiki/proxy.md` for server mode, run modes, endpoint shape, and legacy mode aliases.
22 +- `wiki/ccr.md` for Compress-Cache-Retrieve.
23 +- `wiki/compression.md` for content routing, JSON/code/log/text structure preservation.
24 +- `headroom/compress.py` for the direct Python compression API.
25 +- `headroom/pipeline.py` for pipeline stages.
26 +- `headroom/transforms/content_router.py` for routing concepts.
27 +- `headroom/transforms/smart_crusher.py`, `search_compressor.py`, `log_compressor.py`, and `code_compressor.py` for concrete compression families.
29 +Pi and pi-ext files inspected:
31 +- `/home/me/Workspace/dotfiles/pi-ext/AGENTS.md` for mise-only tasks, compact tool widget rules, cross-platform rules, and markdown style.
32 +- `/home/me/.pi/agent/skills/pi/references/extensions.md` for extension hooks, commands, tools, UI, and provider APIs.
33 +- `/home/me/.pi/agent/skills/pi/references/extension-lifecycle.md` for reload/session safety.
34 +- `/home/me/.pi/agent/skills/pi/references/patterns.md` for custom compaction and hook examples.
35 +- `mise.toml` for required tasks: `mise run fmt`, `lint`, `test`, `check`, `ci`, `readme`, `changelog`.
36 +- root `package.json` for workspace and Pi package exports.
37 +- `extensions/ctx/index.ts`, `README.md`, `package.json`, and `__tests__/ctx.test.ts` for minimal `tool_result` compression.
38 +- `extensions/terse/index.ts`, `README.md`, and `package.json` for deterministic prompt injection.
39 +- `extensions/tldr/index.ts` and tests for bounded UI/widget rendering.
40 +- `extensions/goldfish/index.ts`, `README.md`, and `package.json` for session entries, commands, state replay, and UI state.
41 +- `extensions/builtin-tools/index.ts` for small command/hook style.
42 +- `scripts/extensions.mjs` for extension package checks and README generation expectations.
44 +## Problem
46 +Headroom is a broad context-compression system.
47 +Its strongest ideas are useful for Pi: content-aware compression, reversible retrieval, and visible savings.
48 +Its implementation is not Pi-idiomatic for this repo.
50 +The TypeScript SDK is only an HTTP client.
51 +It calls `POST /v1/compress` on a local Headroom proxy and does not run compression in Node.
52 +The full local implementation lives across Python, Rust, maturin, native extensions, FastAPI, provider shims, optional ML stacks, telemetry, dashboards, memory systems, agent wrappers, and MCP.
53 +That is the wrong shape for `pi-ext`.
55 +`cmprss` should be a Pi extension that hooks Pi itself.
56 +It should compress oversized Pi context before it reaches the model, expose a Pi retrieval tool when exact omitted bytes are needed, and report small bounded savings.
57 +It must not start or require an external server.
59 +## Goals
61 +- Build `cmprss` as a Pi-only TypeScript extension under `extensions/cmprss`.
62 +- Run entirely inside the Pi extension runtime.
63 +- Compress large tool results before they enter model context.
64 +- Preserve enough structure for the model to decide whether to continue or retrieve exact content.
65 +- Provide reversible retrieval for compressed blocks through a Pi tool.
66 +- Keep all state session-local and reload-safe.
67 +- Use deterministic Node/TypeScript code only.
68 +- Allow tiny npm parsing dependencies when they materially improve compression correctness.
69 +- Follow existing extension shape: package, README, tests, small files, no ad-hoc scripts.
70 +- Keep collapsed UI/tool output compact; put verbose lists behind expanded tool rendering if custom rendering is added.
71 +- Stay cross-platform: Node APIs only, no shell helpers.
73 +## Non-goals
75 +- No Headroom proxy.
76 +- No HTTP server.
77 +- No Python runtime.
78 +- No Rust or native extension.
79 +- No `headroom-ai` package dependency.
80 +- No MCP server.
81 +- No large runtime dependency stack.
82 +- No dependency that starts subprocesses, downloads models, opens sockets, or needs native compilation.
83 +- No provider wrapper.
84 +- No `wrap` or `unwrap` commands for Claude, Codex, Cursor, OpenCode, or any other host.
85 +- No dashboard.
86 +- No telemetry, pricing, subscription, quota, OTEL, Langfuse, or data flywheel.
87 +- No cross-agent memory.
88 +- No `learn` feature.
89 +- No ML compression, embeddings, ONNX, PyTorch, HuggingFace, Magika, tree-sitter, ast-grep, or vector DB in v1.
90 +- No legacy Headroom env vars or mode aliases.
91 +- No compatibility with old Headroom config files.
92 +- No mutation of user files.
93 +- No compression of data needed for exact edits unless explicitly recoverable and safe.
95 +## Headroom concepts to keep
97 +### Keep: content routing
99 +Headroom routes content to different compressors.
100 +`cmprss` should keep this idea, but use small deterministic detectors:
102 +- JSON object/array.
103 +- Search result lines.
104 +- Logs/test/build output.
105 +- Generic text.
107 +No ML detector.
108 +No native detector.
109 +No watchdogs.
110 +No fallback maze.
112 +### Keep: structure preservation
114 +The model needs shape, not full bulk.
115 +`cmprss` should preserve keys, filenames, line numbers, error lines, summaries, and head/tail anchors.
117 +### Keep: CCR, but Pi-native
119 +Headroom's CCR idea is good: compressed output includes a retrieval marker and the original remains available.
120 +`cmprss` should implement a small Pi-native version:
122 +- Store the original compressed block under a hash.
123 +- Emit a marker with hash, original size, compressed size, and omitted count.
124 +- Register a `cmprss_retrieve` tool that returns the exact original block by hash.
126 +No injected provider tools.
127 +No response-handler loop.
128 +No automatic second provider call.
129 +Pi already has tools; the agent can call the retrieval tool normally.
131 +### Keep: savings metrics
133 +Report approximate bytes and token estimates.
134 +Use `chars / 4` or UTF-8 bytes / 4.
135 +Do not pretend exact tokenizer parity.
137 +## Headroom concepts to reject
139 +### Reject: proxy mode
141 +Headroom's proxy is the main production path.
142 +It intercepts provider requests and exposes OpenAI/Anthropic-compatible endpoints.
143 +`cmprss` must not do this.
144 +Pi already owns the agent loop and tool lifecycle.
145 +Hooking `tool_result` and optionally `context` is simpler and more direct.
147 +### Reject: TypeScript SDK dependency
149 +The SDK calls the proxy.
150 +Using it would recreate the external server requirement.
152 +### Reject: provider adapters
154 +Pi providers already exist in Pi.
155 +`cmprss` should not register or proxy models.
157 +### Reject: native and ML stack
159 +Headroom's high-end compression depends on Python/Rust/native/ML pieces.
160 +This repo favors small Pi extensions.
161 +The useful first version is deterministic and local.
163 +### Allow: tiny parser dependencies
165 +Tiny npm dependencies are allowed when they reduce brittle parsing code.
166 +They must be pure JavaScript or already-normal Node packages, cross-platform, deterministic, and small.
167 +Good candidates: `diff` for diff parsing and hunk handling, and `jsonc-parser` for tolerant JSON/JSONC parsing.
168 +Possible later candidate: `@babel/parser` for opt-in JavaScript/TypeScript structural summaries.
169 +Reject any dependency that brings ML models, native build chains, network fetches, or broad framework surface.
171 +### Reject: legacy compatibility
173 +No accepted aliases like `token_headroom` or `cost_savings`.
174 +No old Headroom env vars.
175 +No migration paths.
176 +No hidden compatibility shims.
177 +If a setting is wrong, fail clearly or ignore with a bounded warning.
179 +## Product behavior
181 +### Default behavior
183 +When enabled, `cmprss` watches Pi `tool_result` events.
184 +If a result is below the configured byte threshold, it passes through unchanged.
185 +If a result is above threshold and the tool/result class is compressible, it replaces the result content with compressed text and adds structured details.
187 +Default mode: enabled.
188 +Default threshold: `20000` UTF-8 bytes.
189 +Default persisted retrieval store capacity: `64` blocks per session branch.
190 +Default maximum retrieved bytes: unlimited for exact hash retrieval, because retrieval is explicit.
192 +### Compressible tools
194 +Compress by default:
196 +- `bash`
197 +- `grep`
198 +- `find`
199 +- `ls`
200 +- custom tools with text results, if their result text exceeds threshold and they are not marked unsafe
202 +Conservative handling:
204 +- `read` is not compressed when the input has `offset`, `limit`, or `l`.
205 +- `read` full-file output may be compressed only with a marker that instructs the agent to use `read offset/limit` or `cmprss_retrieve` before editing hidden content.
206 +- `edit`, `write`, and mutation result metadata are never compressed.
208 +Rationale: Pi editing depends on exact `oldText` matching.
209 +Do not hide bytes the model may need for an edit unless exact retrieval is obvious.
211 +### Commands
213 +Register `/cmprss`.
215 +Supported subcommands:
217 +```text
218 +/cmprss
219 +/cmprss status
220 +/cmprss on
221 +/cmprss off
222 +/cmprss stats
223 +/cmprss clear
224 +```
226 +Behavior:
228 +- Bare `/cmprss` prints status.
229 +- `status` prints enabled state, threshold, stored block count, and session savings.
230 +- `on` enables compression for the current session branch.
231 +- `off` disables compression for the current session branch.
232 +- `stats` prints compressed block summaries with bounded rows.
233 +- `clear` drops stored originals for the current session branch and resets counters.
235 +Do not add `wrap`, `proxy`, `server`, `dashboard`, `learn`, `doctor`, or `perf`.
237 +### Tool
239 +Register `cmprss_retrieve`.
241 +Parameters:
243 +```ts
245 + hash: string;
247 +```
249 +Result:
251 +- Exact original text for the hash.
252 +- Metadata: hash, original bytes, compressed bytes, compressor, source tool, created time.
254 +If the hash is unknown, throw or return an error result clearly saying `cmprss: unknown hash`.
255 +Do not fake success.
257 +### Markers
259 +Every compressed block includes a visible marker:
261 +```text
262 +[cmprss: <tool> <originalBytes>B → <compressedBytes>B, saved ~<tokens>t, hash=<hash>]
263 +Use cmprss_retrieve(hash="<hash>") for exact original text.
264 +```
266 +For `read`, append:
268 +```text
269 +Before editing hidden content, retrieve exact text or use read offset/limit.
270 +```
272 +### Settings
274 +Pi settings namespace:
276 +```json
278 + "cmprss": {
279 + "enabled": true,
280 + "thresholdBytes": 20000,
281 + "maxStoredBlocks": 64,
282 + "status": true,
283 + "tools": ["bash", "grep", "find", "ls", "read", "custom"]
286 +```
288 +Rules:
290 +- Settings are read from Pi settings through the extension API if available.
291 +- Session commands override settings for the current branch only.
292 +- Invalid values fall back to defaults and may notify once in UI sessions.
293 +- No environment variables in v1.
294 +- No legacy aliases.
296 +### State
298 +Use custom session entries for branch-aware state.
300 +Entry type: `cmprss-state`.
302 +Events:
304 +```ts
305 +type CmprssStateEvent =
306 + | { op: "enabled"; enabled: boolean; at: number }
307 + | { op: "clear"; at: number }
308 + | { op: "block"; meta: CmprssBlockMeta; originalText: string; at: number };
309 +```
311 +The original text is persisted in the custom entry for every compressed block.
312 +For Pi bash results that arrive pre-truncated with `details.fullOutputPath`, persist that file content instead of the visible truncated text.
313 +For v1, persist exact originals in Pi session custom entries so retrieval survives Pi restarts.
314 +On reload/resume, replay metadata and originals from the current branch before accepting `cmprss_retrieve` calls.
315 +When a hash is absent from the replayed branch, `cmprss_retrieve` must fail loud with `cmprss: unknown hash`.
317 +This is intentionally simpler than Headroom's SQLite/Redis memory stack.
318 +It bloats the session file by design, because restart-stable retrieval is required for good results.
320 +Human decision: v1 must persist originals in session history.
321 +No in-memory-only retrieval mode.
323 +### UI
325 +When `cmprss.status` is true and `ctx.hasUI` is true, set a compact status segment:
327 +```text
328 +cmprss: <savedTokens>t
329 +```
331 +When disabled:
333 +```text
334 +cmprss: off
335 +```
337 +Do not add a full dashboard.
338 +Do not add noisy notifications for every compression.
339 +Notify only on first compression in a session and on command changes.
341 +### Details metadata
343 +When a tool result is compressed, merge existing details and add:
345 +```ts
346 +type CmprssDetails = {
347 + cmprss: {
348 + hash: string;
349 + compressor: "json" | "search" | "log" | "text";
350 + originalBytes: number;
351 + compressedBytes: number;
352 + tokensSavedEstimate: number;
353 + sourceTool: string;
354 + };
356 +```
358 +Do not drop existing details.
360 +## Porting priority
362 +Port Headroom compressor ideas in this order:
364 +1. Search compressor.
365 + High ROI, easy TypeScript, preserves paths and line numbers, no dependency needed.
366 +2. Log/test compressor.
367 + Huge savings on failing builds and CI output, deterministic, no dependency needed.
368 +3. JSON/table crusher.
369 + Worth it for tool payloads and API dumps; use `jsonc-parser` only if plain parsing is too brittle.
370 +4. Diff compressor.
371 + Important for pi-ext review workflows; use `diff` if it shrinks hunk parsing and preserves headers reliably.
372 +5. Stacktrace compressor.
373 + Implement as a specialized log path that keeps full causal exception chains.
375 +Defer or skip:
377 +- ML/Kompress/ONNX: too heavy, opaque, slow, and not Pi-extension-shaped.
378 +- Code AST compressor: risky for edit workflows; only consider opt-in later.
379 +- CacheAligner: provider-cache concern, less useful for Pi tool-result compression.
380 +- Memory/vector/TOIN: sprawling and out of scope.
382 +## Compression algorithms
384 +### JSON compressor
386 +Trigger:
388 +- Text parses as JSON object or array.
389 +- Or text contains a JSON-looking top-level array/object after a known tool envelope.
391 +Behavior:
393 +- For arrays: keep first N, last N, and anomalous/error-looking items.
394 +- For objects: preserve keys and short scalar values; summarize long arrays/strings.
395 +- Preserve valid JSON when possible.
396 +- If preserving valid JSON would cost too much code, emit a clear text summary plus JSON snippets.
398 +Default caps:
400 +- Keep first 5 array items.
401 +- Keep last 5 array items.
402 +- Keep up to 10 error/anomaly items.
403 +- Cap rendered JSON at threshold target.
405 +Anomaly hints:
407 +- Keys or values containing `error`, `fail`, `fatal`, `warn`, `exception`, `denied`, `timeout`, `missing`, `deprecated`, `critical`.
408 +- Objects with unique key sets compared to neighbors.
410 +### Search compressor
412 +Trigger lines like:
414 +```text
415 +path/to/file.ts:42:match text
416 +path/to/file.ts-43-context text
417 +```
419 +Behavior:
421 +- Group by file.
422 +- Keep first and last match per file.
423 +- Keep matches with error/anomaly hints.
424 +- Cap files and matches.
425 +- Preserve file paths and line numbers exactly.
427 +Default caps:
429 +- 15 files.
430 +- 5 matches per file.
431 +- 40 total matches.
433 +### Log compressor
435 +Trigger:
437 +- Many lines.
438 +- Contains build/test/log markers such as `ERROR`, `FAIL`, stack traces, assertion failures, `npm ERR`, `pytest`, `cargo`, `warning`, `Traceback`, `Exception`.
440 +Behavior:
442 +- Keep command exit summary when present.
443 +- Keep first 20 lines.
444 +- Keep all high-signal error/failure lines up to cap.
445 +- Keep warning lines up to cap.
446 +- Keep traceback chunks with surrounding context.
447 +- Keep last 40 lines.
448 +- Dedupe repeated low-signal lines.
450 +Default caps:
452 +- 10 error groups.
453 +- 5 warning groups.
454 +- 3 stack traces.
455 +- 120 total lines.
457 +### Diff compressor
459 +Trigger:
461 +- Unified diff text.
462 +- Git diff output from `bash` or review tools.
464 +Behavior:
466 +- Preserve file headers exactly.
467 +- Preserve hunk headers exactly.
468 +- Keep changed lines.
469 +- Keep conflict markers and lines containing error/anomaly hints.
470 +- Drop large unchanged context blocks first.
471 +- Emit omitted context counts per file or hunk.
473 +Dependency option:
475 +- Use `diff` if it makes parsing smaller and more reliable than handwritten hunk parsing.
477 +### Stacktrace compressor
479 +Trigger:
481 +- Python tracebacks.
482 +- JavaScript/TypeScript stack traces.
483 +- Java/Kotlin stack traces.
484 +- Rust panic traces.
485 +- Go panic traces.
487 +Behavior:
489 +- Keep exception type and message.
490 +- Keep first and last frame.
491 +- Keep application frames over dependency frames.
492 +- Keep chained causes: `caused by`, `During handling`, `The above exception`, `Suppressed`.
493 +- Compress repeated dependency frames with counts.
495 +### Text compressor
497 +Fallback.
499 +Behavior:
501 +- Keep head and tail.
502 +- Include omitted line/byte counts.
503 +- Preserve code fences if they begin or end in kept windows.
504 +- Never invent summary text beyond counts and marker.
506 +Default:
508 +- First 40 lines.
509 +- Last 80 lines.
511 +## File structure
513 +```text
514 +extensions/cmprss/
515 +├── index.ts
516 +├── package.json
517 +├── README.md
518 +├── src/
519 +│ ├── commands.ts
520 +│ ├── compressors.ts
521 +│ ├── config.ts
522 +│ ├── content.ts
523 +│ ├── state.ts
524 +│ └── ui.ts
525 +└── __tests__/
526 + ├── compressors.test.ts
527 + ├── config.test.ts
528 + ├── harness.test.ts
529 + └── state.test.ts
530 +```
532 +Keep files small.
533 +No generated code.
534 +No build step.
536 +## Dependency policy
538 +Allowed npm deps must be small, deterministic, cross-platform, and compression-specific.
539 +Prefer no dep when plain TypeScript is clearer.
540 +Use a dep only when it improves correctness enough to shrink code.
542 +Approved initial candidates:
544 +- `diff` for unified diff parsing and hunk selection.
545 +- `jsonc-parser` for tolerant JSON/JSONC parsing if plain `JSON.parse` misses common tool output.
547 +Deferred candidate:
549 +- `@babel/parser` for opt-in JS/TS AST summaries after v1 proves value.
551 +Banned deps:
553 +- ML/model runtimes.
554 +- Native build requirements.
555 +- Network clients for compression.
556 +- Frameworks.
557 +- Headroom SDK/client packages.
559 +## Package shape
561 +`extensions/cmprss/package.json`:
563 +- `name`: `@bugabinga/pi-ext-cmprss`
564 +- `type`: `module`
565 +- `main`: `index.ts`
566 +- `private`: `true`
567 +- peer dependencies only for Pi packages and `typebox` if the retrieve tool uses TypeBox schemas
568 +- dev dependencies: `vitest`, `@marcfargas/pi-test-harness`
570 +Root `package.json` already exports `extensions/*/index.ts`, so no root package change is required for extension discovery unless a skill is later added.
571 +No skills are needed in v1.
573 +## Hook architecture
575 +Register:
577 +- `session_start` to replay session state and set status.
578 +- `session_shutdown` to clear process-local originals for the old runtime.
579 +- `tool_result` to compress eligible text results.
580 +- `cmprss_retrieve` tool.
581 +- `/cmprss` command.
583 +Do not capture stale `ctx`.
584 +Keep process globals reload-safe or clear them on `session_shutdown`.
586 +## Error handling
588 +- Compression failure returns the original result unchanged and records a bounded warning in details.
589 +- Retrieval miss fails loud.
590 +- Invalid command arguments print help.
591 +- Invalid settings fall back to defaults.
592 +- Aborted tool calls respect `signal.aborted` where available.
593 +- Hash collisions are practically ignored with SHA-256 truncation, but store compares full hash internally.
595 +No silent lossy fallback for retrieval.
596 +No background workers.
597 +No timers.
599 +## Security and secret handling
601 +`cmprss` stores exact tool output in process memory.
602 +That may include secrets already exposed to the model by a tool result.
603 +It must not print stored originals in stats, UI, or details.
604 +Only `cmprss_retrieve(hash)` returns the exact original.
606 +Do not scan files independently in v1.
607 +Only handle Pi tool outputs already produced.
609 +## Testing
611 +Use `mise run test -- cmprss` or the repo's normal test task.
612 +Do not bypass mise for listed repo tasks.
614 +Required tests:
616 +- Harness registers one command, one tool, and expected hooks.
617 +- Small results pass through unchanged.
618 +- Disabled mode passes through unchanged.
619 +- Oversized text compresses and stores retrievable original.
620 +- Retrieval returns exact original for a known hash.
621 +- Retrieval miss fails loud.
622 +- Existing details are preserved.
623 +- Explicit `read` windows are not compressed.
624 +- Mutation tools are not compressed.
625 +- JSON array keeps anchors and emits counts.
626 +- Search output preserves paths and line numbers.
627 +- Log output preserves error/traceback/failure lines.
628 +- Session replay applies latest enable/disable entry.
629 +- Clear tombstones prior persisted originals and resets counters.
630 +- Restart/replay restores `cmprss_retrieve` for prior compressed blocks.
632 +Required checks before implementation is considered done:
634 +```text
635 +mise run fmt
636 +mise run lint
637 +mise run test
638 +mise run check
639 +```
641 +Full release gate:
643 +```text
644 +mise run ci
645 +```
647 +## README requirements
649 +`extensions/cmprss/README.md` should include:
651 +- Load through root `pi-ext` package.
652 +- Commands/tools/settings/hooks list.
653 +- Compression safety rules.
654 +- Retrieval example.
655 +- Statement that no server, proxy, Python, Rust, ML, MCP, telemetry, or dashboard exists.
657 +## Acceptance checks
659 +Observable acceptance for the spec:
661 +- The design names the exact Headroom concepts retained.
662 +- The design names the exact Headroom concepts rejected.
663 +- The design avoids external server/proxy architecture.
664 +- The design uses Pi hooks/tools/commands/session state directly.
665 +- The design identifies existing pi-ext conventions inspected.
666 +- The design includes settings, command surface, retrieval tool, state model, UI, tests, and release checks.
667 +- The design records the user decision that originals must persist across Pi restarts.
669 +Observable acceptance for future implementation:
671 +- Loading the extension does not start a server or spawn a process.
672 +- `cmprss_retrieve` works without network access.
673 +- A large `bash` output is compressed before the next model turn.
674 +- The original large output can be retrieved by hash after restarting Pi.
675 +- Explicit `read offset/limit` output is never compressed.
676 +- `mise run ci` passes.
678 +## Decided tradeoff
680 +V1 persists exact originals in Pi session history.
681 +This makes compressed blocks restart-stable and branch-local.
682 +The accepted cost is larger session files.

extensions/cmprss/README.md

Added
new file mode 100644
1 +# cmprss
3 +Compresses oversized Pi tool results and persists exact originals for retrieval.
5 +## Install / load
7 +Loaded through the root `pi-ext` package.
8 +See [../../README.md](../../README.md).
10 +## Commands / tools / settings
12 +Pi commands:
14 +- `/cmprss` — show status.
15 +- `/cmprss status` — show status.
16 +- `/cmprss stats` — show stored block count and estimated savings.
17 +- `/cmprss on` — enable compression for the current session branch.
18 +- `/cmprss off` — disable compression for the current session branch.
19 +- `/cmprss clear` — tombstone stored originals for the current session branch.
21 +Pi tools:
23 +- `cmprss_retrieve` — retrieve exact original text by hash.
25 +Pi hooks:
27 +- `session_start` — replay persisted originals and state.
28 +- `session_shutdown` — clear process-local cache.
29 +- `tool_result` — compress eligible oversized tool output.
31 +Pi settings:
33 +```json
35 + "cmprss": {
36 + "enabled": true,
37 + "thresholdBytes": 20000,
38 + "maxStoredBlocks": 64,
39 + "status": true,
40 + "tools": ["bash", "grep", "find", "ls", "read"]
41 + }
43 +```
45 +## Behavior
47 +`cmprss` compresses large text tool results before the next model turn.
48 +It stores the exact original in Pi session history, so restarting Pi can still retrieve the hidden bytes.
49 +For pre-truncated Pi bash results, it persists `details.fullOutputPath` content when available.
51 +Default compressors:
53 +- JSON arrays and objects.
54 +- grep/ripgrep-style search output.
55 +- unified diffs.
56 +- stack traces.
57 +- build/test logs.
58 +- generic head/tail text.
60 +Safety rules:
62 +- Never compresses `edit` or `write` results.
63 +- Never compresses `read` output when `offset`, `limit`, or `l` is set.
64 +- Preserves existing tool result details and adds `details.cmprss`.
65 +- Retrieval misses fail loudly with `cmprss: unknown hash`.
67 +Example marker:
69 +```text
70 +[cmprss: bash 58000B → 7000B, saved ~12000t, hash=abc123]
71 +Use cmprss_retrieve(hash="abc123") for exact original text.
72 +```
74 +No server.
75 +No proxy.
76 +No Python.
77 +No Rust.
78 +No ML.
79 +No MCP.
80 +No telemetry.
81 +No dashboard.

extensions/cmprss/__tests__/cmprss.test.ts

Added
new file mode 100644
1 +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
2 +import { tmpdir } from "node:os";
3 +import { join } from "node:path";
4 +import { describe, expect, test } from "vitest";
5 +import cmprssExtension, { __test } from "../index.ts";
7 +const {
8 + DEFAULT_CONFIG,
9 + compressToolResult,
10 + createRuntime,
11 + replayState,
12 + runCommand,
13 +} = __test;
15 +function bigText(lines = 200): string {
16 + return Array.from({ length: lines }, (_, i) => `line ${i + 1}`).join("\n");
19 +describe("cmprss extension", () => {
20 + test("registers command, retrieve tool, and Pi hooks", () => {
21 + const events: string[] = [];
22 + const tools: string[] = [];
23 + const commands: string[] = [];
24 + cmprssExtension({
25 + on(name: string) {
26 + events.push(name);
27 + },
28 + registerTool(tool: { name: string }) {
29 + tools.push(tool.name);
30 + },
31 + registerCommand(name: string) {
32 + commands.push(name);
33 + },
34 + appendEntry() {},
35 + } as any);
37 + expect(events).toEqual([
38 + "session_start",
39 + "session_shutdown",
40 + "tool_result",
41 + ]);
42 + expect(tools).toEqual(["cmprss_retrieve"]);
43 + expect(commands).toEqual(["cmprss"]);
44 + });
46 + test("leaves small output unchanged", () => {
47 + const runtime = createRuntime(DEFAULT_CONFIG);
48 + const result = compressToolResult(runtime, {
49 + toolName: "bash",
50 + input: { command: "echo hi" },
51 + content: [{ type: "text", text: "hi" }],
52 + details: { exitCode: 0 },
53 + isError: false,
54 + });
56 + expect(result).toBeUndefined();
57 + expect(runtime.stats.blocks).toBe(0);
58 + });
60 + test("uses fullOutputPath as persisted original when Pi pre-truncated bash", () => {
61 + const dir = mkdtempSync(join(tmpdir(), "cmprss-"));
62 + try {
63 + const full = bigText(6000);
64 + const shown = full.split("\n").slice(-2000).join("\n");
65 + const fullOutputPath = join(dir, "out.log");
66 + writeFileSync(fullOutputPath, full);
67 + const runtime = createRuntime({ ...DEFAULT_CONFIG, thresholdBytes: 200 });
69 + const result = compressToolResult(runtime, {
70 + toolName: "bash",
71 + input: {},
72 + content: [{ type: "text", text: shown }],
73 + details: { fullOutputPath, truncation: { truncated: true } },
74 + isError: false,
75 + });
77 + const hash = (result?.details as any).cmprss.hash;
78 + expect(runtime.retrieve(hash)).toBe(full);
79 + expect(readFileSync(fullOutputPath, "utf8")).toBe(full);
80 + } finally {
81 + rmSync(dir, { recursive: true, force: true });
82 + }
83 + });
85 + test("compresses oversized output and retrieves exact original", () => {
86 + const runtime = createRuntime({ ...DEFAULT_CONFIG, thresholdBytes: 200 });
87 + const original = bigText();
88 + const result = compressToolResult(runtime, {
89 + toolName: "bash",
90 + input: { command: "make test" },
91 + content: [{ type: "text", text: original }],
92 + details: { exitCode: 1 },
93 + isError: false,
94 + });
96 + expect(result?.content?.[0]?.text).toContain("[cmprss: bash");
97 + expect(result?.content?.[0]?.text).toContain("cmprss_retrieve");
98 + expect(result?.details).toMatchObject({ exitCode: 1 });
99 + const hash = (result?.details as any).cmprss.hash;
100 + expect(runtime.retrieve(hash)).toBe(original);
101 + expect(runtime.entries[0]).toMatchObject({
102 + op: "block",
103 + originalText: original,
104 + });
105 + });
107 + test("replay restores persisted originals after restart", () => {
108 + const first = createRuntime({ ...DEFAULT_CONFIG, thresholdBytes: 200 });
109 + const original = bigText();
110 + const result = compressToolResult(first, {
111 + toolName: "bash",
112 + input: { command: "logs" },
113 + content: [{ type: "text", text: original }],
114 + isError: false,
115 + });
116 + const hash = (result?.details as any).cmprss.hash;
118 + const second = createRuntime(DEFAULT_CONFIG);
119 + replayState(
120 + second,
121 + first.entries.map((data) => ({
122 + type: "custom",
123 + customType: "cmprss-state",
124 + data,
125 + })),
126 + );
128 + expect(second.retrieve(hash)).toBe(original);
129 + });
131 + test("clear tombstones prior persisted originals", () => {
132 + const runtime = createRuntime({ ...DEFAULT_CONFIG, thresholdBytes: 200 });
133 + const result = compressToolResult(runtime, {
134 + toolName: "bash",
135 + input: {},
136 + content: [{ type: "text", text: bigText() }],
137 + isError: false,
138 + });
139 + const hash = (result?.details as any).cmprss.hash;
141 + runCommand(runtime, "clear");
143 + expect(() => runtime.retrieve(hash)).toThrow("cmprss: unknown hash");
144 + expect(runtime.stats.blocks).toBe(0);
145 + });
147 + test("does not compress custom tools unless configured", () => {
148 + const runtime = createRuntime({ ...DEFAULT_CONFIG, thresholdBytes: 10 });
149 + expect(
150 + compressToolResult(runtime, {
151 + toolName: "web_fetch",
152 + input: {},
153 + content: [{ type: "text", text: bigText() }],
154 + isError: false,
155 + }),
156 + ).toBeUndefined();
158 + const configured = createRuntime({
159 + ...DEFAULT_CONFIG,
160 + thresholdBytes: 10,
161 + tools: [...DEFAULT_CONFIG.tools, "web_fetch"],
162 + });
163 + expect(
164 + compressToolResult(configured, {
165 + toolName: "web_fetch",
166 + input: {},
167 + content: [{ type: "text", text: bigText() }],
168 + isError: false,
169 + }),
170 + ).toBeDefined();
171 + });
173 + test("does not compress explicit read windows or mutation tools", () => {
174 + const runtime = createRuntime({ ...DEFAULT_CONFIG, thresholdBytes: 10 });
175 + for (const event of [
177 + toolName: "read",
178 + input: { path: "x", offset: 10 },
179 + content: [{ type: "text", text: bigText() }],
180 + isError: false,
181 + },
183 + toolName: "read",
184 + input: { path: "x", limit: 10 },
185 + content: [{ type: "text", text: bigText() }],
186 + isError: false,
187 + },
189 + toolName: "edit",
190 + input: {},
191 + content: [{ type: "text", text: bigText() }],
192 + isError: false,
193 + },
195 + toolName: "write",
196 + input: {},
197 + content: [{ type: "text", text: bigText() }],
198 + isError: false,
199 + },
200 + ]) {
201 + expect(compressToolResult(runtime, event as any)).toBeUndefined();
203 + });
205 + test("json array compression keeps anchors and error items", () => {
206 + const runtime = createRuntime({ ...DEFAULT_CONFIG, thresholdBytes: 80 });
207 + const rows = Array.from({ length: 40 }, (_, i) => ({
208 + id: i + 1,
209 + status: "ok",
210 + }));
211 + rows[20] = { id: 21, status: "fatal error" };
212 + const result = compressToolResult(runtime, {
213 + toolName: "bash",
214 + input: {},
215 + content: [{ type: "text", text: JSON.stringify(rows) }],
216 + isError: false,
217 + });
218 + const text = result?.content?.[0]?.text ?? "";
220 + expect(text).toContain('"id": 1');
221 + expect(text).toContain('"id": 40');
222 + expect(text).toContain("fatal error");
223 + expect(text).toContain("json array: 40 items");
224 + });
226 + test("search compression preserves paths and line numbers", () => {
227 + const runtime = createRuntime({ ...DEFAULT_CONFIG, thresholdBytes: 80 });
228 + const text = Array.from(
229 + { length: 80 },
230 + (_, i) => `src/a.ts:${i + 1}:match ${i + 1}`,
231 + ).join("\n");
232 + const result = compressToolResult(runtime, {
233 + toolName: "grep",
234 + input: {},
235 + content: [{ type: "text", text }],
236 + isError: false,
237 + });
239 + expect(result?.content?.[0]?.text).toContain("src/a.ts:1:match 1");
240 + expect(result?.content?.[0]?.text).toContain("src/a.ts:80:match 80");
241 + expect(result?.content?.[0]?.text).toContain("search: 80 matches");
242 + });
244 + test("stacktrace compression wins before search-like frame paths", () => {
245 + const runtime = createRuntime({ ...DEFAULT_CONFIG, thresholdBytes: 80 });
246 + const text = [
247 + "Error: top",
248 + ...Array.from(
249 + { length: 300 },
250 + (_, i) => ` at fn${i} (/dep/file${i}.js:${i}:1)`,
251 + ),
252 + "Caused by: Error: root",
253 + ].join("\n");
254 + const result = compressToolResult(runtime, {
255 + toolName: "bash",
256 + input: {},
257 + content: [{ type: "text", text }],
258 + isError: false,
259 + });
261 + expect((result?.details as any).cmprss.compressor).toBe("stacktrace");
262 + expect(result?.content?.[0]?.text).toContain("Caused by: Error: root");
263 + });
265 + test("log compression preserves failures and tail", () => {
266 + const runtime = createRuntime({ ...DEFAULT_CONFIG, thresholdBytes: 80 });
267 + const text = [
268 + ...Array.from({ length: 50 }, (_, i) => `noise ${i}`),
269 + "ERROR build failed",
270 + "Traceback (most recent call last):",
271 + "Exception: boom",
272 + ...Array.from({ length: 50 }, (_, i) => `tail ${i}`),
273 + ].join("\n");
274 + const result = compressToolResult(runtime, {
275 + toolName: "bash",
276 + input: {},
277 + content: [{ type: "text", text }],
278 + isError: false,
279 + });
281 + expect(result?.content?.[0]?.text).toContain("ERROR build failed");
282 + expect(result?.content?.[0]?.text).toContain("Traceback");
283 + expect(result?.content?.[0]?.text).toContain("tail 49");
284 + });
285 +});

extensions/cmprss/__tests__/harness.test.ts

Added
new file mode 100644
1 +import { describe, expect, test } from "vitest";
2 +import cmprssExtension from "../index.ts";
4 +describe("cmprss harness", () => {
5 + test("registers Pi surfaces", () => {
6 + const events: string[] = [];
7 + const tools: string[] = [];
8 + const commands: string[] = [];
10 + cmprssExtension({
11 + on(name: string) {
12 + events.push(name);
13 + },
14 + registerTool(tool: { name: string }) {
15 + tools.push(tool.name);
16 + },
17 + registerCommand(name: string) {
18 + commands.push(name);
19 + },
20 + appendEntry() {},
21 + } as any);
23 + expect(events).toEqual([
24 + "session_start",
25 + "session_shutdown",
26 + "tool_result",
27 + ]);
28 + expect(tools).toEqual(["cmprss_retrieve"]);
29 + expect(commands).toEqual(["cmprss"]);
30 + });
31 +});

extensions/cmprss/index.ts

Added
new file mode 100644
1 +import { createHash } from "node:crypto";
2 +import { readFileSync } from "node:fs";
3 +import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
4 +import { Type } from "typebox";
6 +const ENTRY_TYPE = "cmprss-state";
7 +const STATUS_KEY = "cmprss";
9 +const ERROR_RE =
10 + /\b(error|fail|fatal|warn|exception|denied|timeout|missing|deprecated|critical|traceback|panic)\b/i;
11 +const SEARCH_RE =
12 + /^(?<file>.+?)(?<sep>[:-])(?<line>\d+)(?<sep2>[:-])(?<body>.*)$/;
13 +const STACK_RE =
14 + /\b(Traceback \(most recent call last\)|Exception:|Caused by:|Error:|panic:|at \S+ \(|\s+at \S+)/;
16 +type TextContent = { type?: unknown; text?: unknown };
17 +type ToolResultEventLike = {
18 + toolName?: unknown;
19 + input?: Record<string, unknown>;
20 + content?: unknown;
21 + details?: unknown;
22 + isError?: unknown;
23 +};
24 +type CmprssConfig = {
25 + enabled: boolean;
26 + thresholdBytes: number;
27 + maxStoredBlocks: number;
28 + status: boolean;
29 + tools: string[];
30 +};
31 +type Compressor = "json" | "search" | "log" | "text" | "diff" | "stacktrace";
32 +type BlockMeta = {
33 + hash: string;
34 + compressor: Compressor;
35 + originalBytes: number;
36 + compressedBytes: number;
37 + tokensSavedEstimate: number;
38 + sourceTool: string;
39 + createdAt: number;
40 +};
41 +type StateEvent =
42 + | { op: "enabled"; enabled: boolean; at: number }
43 + | { op: "clear"; at: number }
44 + | { op: "block"; meta: BlockMeta; originalText: string; at: number };
45 +type Runtime = {
46 + config: CmprssConfig;
47 + enabled: boolean;
48 + blocks: Map<string, { meta: BlockMeta; originalText: string }>;
49 + entries: StateEvent[];
50 + stats: { blocks: number; originalBytes: number; compressedBytes: number };
51 + append?: (event: StateEvent) => void;
52 + retrieve(hash: string): string;
53 +};
55 +type Compression = {
56 + compressor: Compressor;
57 + text: string;
58 +};
60 +export const DEFAULT_CONFIG: CmprssConfig = {
61 + enabled: true,
62 + thresholdBytes: 20_000,
63 + maxStoredBlocks: 64,
64 + status: true,
65 + tools: ["bash", "grep", "find", "ls", "read"],
66 +};
68 +function isRecord(value: unknown): value is Record<string, unknown> {
69 + return value !== null && typeof value === "object" && !Array.isArray(value);
72 +function fullOutputPath(details: unknown): string | undefined {
73 + return isRecord(details) && typeof details.fullOutputPath === "string"
74 + ? details.fullOutputPath
75 + : undefined;
78 +function textOf(content: unknown): string {
79 + if (typeof content === "string") return content;
80 + if (!Array.isArray(content)) return String(content ?? "");
81 + return (content as TextContent[])
82 + .map((part) =>
83 + part?.type === "text" && typeof part.text === "string" ? part.text : "",
84 + )
85 + .filter(Boolean)
86 + .join("\n");
89 +function bytes(text: string): number {
90 + return Buffer.byteLength(text, "utf8");
93 +function readPersistedOutput(path: string): string {
94 + try {
95 + return readFileSync(path, "utf8");
96 + } catch {
97 + return "";
98 + }
101 +function hashText(text: string): string {
102 + return createHash("sha256").update(text).digest("hex").slice(0, 16);
105 +function uniq<T>(items: T[]): T[] {
106 + return [...new Set(items)];
109 +function clipLines(lines: string[], first: number, last: number): string[] {
110 + if (lines.length <= first + last) return lines;
111 + return [
112 + ...lines.slice(0, first),
113 + `[cmprss omitted ${lines.length - first - last} lines]`,
114 + ...lines.slice(-last),
115 + ];
118 +function containsError(value: unknown): boolean {
119 + return ERROR_RE.test(JSON.stringify(value));
122 +function formatJson(value: unknown): string {
123 + return JSON.stringify(value, null, 2);
126 +function compressJson(text: string): Compression | undefined {
127 + let parsed: unknown;
128 + try {
129 + parsed = JSON.parse(text);
130 + } catch {
131 + return undefined;
133 + if (Array.isArray(parsed)) {
134 + const first = parsed.slice(0, 5);
135 + const last = parsed.slice(-5);
136 + const errors = parsed.filter(containsError).slice(0, 10);
137 + const kept = uniq([...first, ...errors, ...last]);
138 + return {
139 + compressor: "json",
140 + text: [
141 + `[cmprss json array: ${parsed.length} items, kept ${kept.length}]`,
142 + formatJson(kept),
143 + ].join("\n"),
144 + };
146 + if (isRecord(parsed)) {
147 + const out: Record<string, unknown> = {};
148 + for (const [key, value] of Object.entries(parsed)) {
149 + if (typeof value === "string" && value.length > 160) {
150 + out[key] =
151 + `${value.slice(0, 100)} …[cmprss ${value.length - 140} chars]… ${value.slice(-40)}`;
152 + } else if (Array.isArray(value) && value.length > 12) {
153 + out[key] = [
154 + ...value.slice(0, 5),
155 + `[cmprss omitted ${value.length - 10} items]`,
156 + ...value.slice(-5),
157 + ];
158 + } else out[key] = value;
160 + return {
161 + compressor: "json",
162 + text: `[cmprss json object]\n${formatJson(out)}`,
163 + };
165 + return undefined;
168 +function parseSearchLine(line: string) {
169 + const match = SEARCH_RE.exec(line);
170 + if (!match?.groups) return undefined;
171 + return {
172 + file: match.groups.file,
173 + line: Number(match.groups.line),
174 + body: match.groups.body,
175 + raw: line,
176 + };
179 +function compressSearch(text: string): Compression | undefined {
180 + const lines = text.split(/\r?\n/);
181 + const matches = lines.map(parseSearchLine).filter((x) => x !== undefined);
182 + if (matches.length < 10 || matches.length < lines.length * 0.6)
183 + return undefined;
184 + const byFile = new Map<string, typeof matches>();
185 + for (const match of matches) {
186 + const group = byFile.get(match.file) ?? [];
187 + group.push(match);
188 + byFile.set(match.file, group);
190 + const kept: string[] = [];
191 + let files = 0;
192 + for (const [file, group] of byFile) {
193 + if (files++ >= 15) break;
194 + kept.push(`[cmprss file: ${file}, ${group.length} matches]`);
195 + const selected = uniq([
196 + ...group.slice(0, 2),
197 + ...group.filter((m) => ERROR_RE.test(m.body)).slice(0, 5),
198 + ...group.slice(-2),
199 + ]);
200 + for (const match of selected.slice(0, 5)) kept.push(match.raw);
202 + return {
203 + compressor: "search",
204 + text: [
205 + `[cmprss search: ${matches.length} matches in ${byFile.size} files]`,
206 + ...kept,
207 + ].join("\n"),
208 + };
211 +function compressDiff(text: string): Compression | undefined {
212 + const lines = text.split(/\r?\n/);
213 + const diffish = lines.filter((line) =>
214 + /^(diff --git|@@ |--- |\+\+\+ |[+-])/.test(line),
215 + ).length;
216 + if (diffish < 20) return undefined;
217 + const kept = lines.filter(
218 + (line) =>
219 + /^(diff --git|@@ |--- |\+\+\+ |[+-])/.test(line) || ERROR_RE.test(line),
220 + );
221 + return {
222 + compressor: "diff",
223 + text: [
224 + `[cmprss diff: ${lines.length} lines, kept ${kept.length}]`,
225 + ...clipLines(kept, 120, 80),
226 + ].join("\n"),
227 + };
230 +function compressStack(text: string): Compression | undefined {
231 + const lines = text.split(/\r?\n/);
232 + const stack = lines.filter((line) => STACK_RE.test(line));
233 + if (stack.length < 4) return undefined;
234 + const kept = lines.filter(
235 + (line, index) =>
236 + STACK_RE.test(line) ||
237 + ERROR_RE.test(line) ||
238 + index < 5 ||
239 + index >= lines.length - 10,
240 + );
241 + return {
242 + compressor: "stacktrace",
243 + text: [
244 + `[cmprss stacktrace: ${lines.length} lines, kept ${kept.length}]`,
245 + ...clipLines(uniq(kept), 80, 80),
246 + ].join("\n"),
247 + };
250 +function compressLog(text: string): Compression | undefined {
251 + const lines = text.split(/\r?\n/);
252 + const signal = lines.filter(
253 + (line) => ERROR_RE.test(line) || STACK_RE.test(line),
254 + );
255 + if (lines.length < 80 || signal.length === 0) return undefined;
256 + const kept = uniq([
257 + ...lines.slice(0, 20),
258 + ...lines.filter((line) => ERROR_RE.test(line)).slice(0, 40),
259 + ...lines.filter((line) => STACK_RE.test(line)).slice(0, 60),
260 + ...lines.slice(-40),
261 + ]);
262 + return {
263 + compressor: "log",
264 + text: [
265 + `[cmprss log: ${lines.length} lines, kept ${kept.length}]`,
266 + ...clipLines(kept, 90, 90),
267 + ].join("\n"),
268 + };
271 +function compressText(text: string): Compression {
272 + const lines = text.split(/\r?\n/);
273 + return {
274 + compressor: "text",
275 + text: [
276 + `[cmprss text: ${lines.length} lines]`,
277 + ...clipLines(lines, 40, 80),
278 + ].join("\n"),
279 + };
282 +function compressContent(text: string): Compression {
283 + return (
284 + compressJson(text) ??
285 + compressDiff(text) ??
286 + compressStack(text) ??
287 + compressSearch(text) ??
288 + compressLog(text) ??
289 + compressText(text)
290 + );
293 +function normalizeConfig(raw: unknown): CmprssConfig {
294 + const cfg = isRecord(raw) ? raw : {};
295 + return {
296 + enabled:
297 + typeof cfg.enabled === "boolean" ? cfg.enabled : DEFAULT_CONFIG.enabled,
298 + thresholdBytes:
299 + typeof cfg.thresholdBytes === "number" &&
300 + Number.isFinite(cfg.thresholdBytes) &&
301 + cfg.thresholdBytes >= 0
302 + ? Math.trunc(cfg.thresholdBytes)
303 + : DEFAULT_CONFIG.thresholdBytes,
304 + maxStoredBlocks:
305 + typeof cfg.maxStoredBlocks === "number" &&
306 + Number.isFinite(cfg.maxStoredBlocks) &&
307 + cfg.maxStoredBlocks > 0
308 + ? Math.trunc(cfg.maxStoredBlocks)
309 + : DEFAULT_CONFIG.maxStoredBlocks,
310 + status:
311 + typeof cfg.status === "boolean" ? cfg.status : DEFAULT_CONFIG.status,
312 + tools: Array.isArray(cfg.tools)
313 + ? cfg.tools.filter((x) => typeof x === "string")
314 + : DEFAULT_CONFIG.tools,
315 + };
318 +function settingsFrom(pi: unknown): CmprssConfig {
319 + const raw = (
320 + pi as {
321 + settingsManager?: {
322 + getSettings?: () => Record<string, unknown> | undefined;
323 + };
325 + ).settingsManager?.getSettings?.();
326 + return normalizeConfig(isRecord(raw?.cmprss) ? raw.cmprss : {});
329 +export function createRuntime(
330 + config: CmprssConfig,
331 + append?: (event: StateEvent) => void,
332 +): Runtime {
333 + const runtime: Runtime = {
334 + config,
335 + enabled: config.enabled,
336 + blocks: new Map(),
337 + entries: [],
338 + stats: { blocks: 0, originalBytes: 0, compressedBytes: 0 },
339 + append,
340 + retrieve(hash: string) {
341 + const block = runtime.blocks.get(hash);
342 + if (!block) throw new Error("cmprss: unknown hash");
343 + return block.originalText;
344 + },
345 + };
346 + return runtime;
349 +function record(runtime: Runtime, event: StateEvent): void {
350 + applyStateEvent(runtime, event);
351 + runtime.entries.push(event);
352 + runtime.append?.(event);
355 +function applyStateEvent(runtime: Runtime, event: StateEvent): void {
356 + if (event.op === "enabled") {
357 + runtime.enabled = event.enabled;
358 + return;
360 + if (event.op === "clear") {
361 + runtime.blocks.clear();
362 + runtime.stats = { blocks: 0, originalBytes: 0, compressedBytes: 0 };
363 + return;
365 + const { meta, originalText } = event;
366 + runtime.blocks.set(meta.hash, { meta, originalText });
367 + while (runtime.blocks.size > runtime.config.maxStoredBlocks) {
368 + const first = runtime.blocks.keys().next().value;
369 + if (!first) break;
370 + runtime.blocks.delete(first);
372 + runtime.stats.blocks = runtime.blocks.size;
373 + runtime.stats.originalBytes += meta.originalBytes;
374 + runtime.stats.compressedBytes += meta.compressedBytes;
377 +function parseStateEvent(data: unknown): StateEvent | undefined {
378 + if (
379 + !isRecord(data) ||
380 + typeof data.op !== "string" ||
381 + typeof data.at !== "number"
383 + return undefined;
384 + if (data.op === "enabled" && typeof data.enabled === "boolean")
385 + return { op: "enabled", enabled: data.enabled, at: data.at };
386 + if (data.op === "clear") return { op: "clear", at: data.at };
387 + if (
388 + data.op === "block" &&
389 + isRecord(data.meta) &&
390 + typeof data.originalText === "string"
392 + return data as StateEvent;
393 + return undefined;
396 +export function replayState(
397 + runtime: Runtime,
398 + entries: Array<{ type?: unknown; customType?: unknown; data?: unknown }>,
399 +): void {
400 + applyStateEvent(runtime, { op: "clear", at: Date.now() });
401 + for (const entry of entries) {
402 + if (entry.type !== "custom" || entry.customType !== ENTRY_TYPE) continue;
403 + const event = parseStateEvent(entry.data);
404 + if (event) applyStateEvent(runtime, event);
408 +function compressible(runtime: Runtime, event: ToolResultEventLike): boolean {
409 + if (!runtime.enabled || event.isError) return false;
410 + const tool = String(event.toolName ?? "");
411 + if (["edit", "write"].includes(tool)) return false;
412 + if (
413 + tool === "read" &&
414 + (event.input?.offset !== undefined ||
415 + event.input?.limit !== undefined ||
416 + event.input?.l !== undefined)
418 + return false;
419 + if (
420 + !runtime.config.tools.includes(tool) &&
421 + !runtime.config.tools.includes("custom")
423 + return false;
424 + return true;
427 +export function compressToolResult(
428 + runtime: Runtime,
429 + event: ToolResultEventLike,
430 +) {
431 + if (!compressible(runtime, event)) return undefined;
432 + const visibleText = textOf(event.content);
433 + const path = fullOutputPath(event.details);
434 + const persistedText = path ? readPersistedOutput(path) : "";
435 + const originalText = persistedText || visibleText;
436 + const originalBytes = bytes(originalText);
437 + if (originalBytes <= runtime.config.thresholdBytes) return undefined;
438 + const compression = compressContent(originalText);
439 + const hash = hashText(originalText);
440 + const marker = `[cmprss: ${String(event.toolName)} ${originalBytes}B → ${bytes(compression.text)}B, saved ~${Math.max(0, Math.floor((originalBytes - bytes(compression.text)) / 4))}t, hash=${hash}]\nUse cmprss_retrieve(hash="${hash}") for exact original text.`;
441 + const readWarning =
442 + event.toolName === "read"
443 + ? "\nBefore editing hidden content, retrieve exact text or use read offset/limit."
444 + : "";
445 + const text = `${marker}${readWarning}\n\n${compression.text}`;
446 + const meta: BlockMeta = {
447 + hash,
448 + compressor: compression.compressor,
449 + originalBytes,
450 + compressedBytes: bytes(text),
451 + tokensSavedEstimate: Math.max(
452 + 0,
453 + Math.floor((originalBytes - bytes(text)) / 4),
454 + ),
455 + sourceTool: String(event.toolName),
456 + createdAt: Date.now(),
457 + };
458 + record(runtime, { op: "block", meta, originalText, at: meta.createdAt });
459 + return {
460 + content: [{ type: "text", text }],
461 + details: {
462 + ...(isRecord(event.details) ? event.details : {}),
463 + cmprss: meta,
464 + },
465 + isError: false,
466 + };
469 +export function runCommand(runtime: Runtime, args: string): string {
470 + const command = args.trim() || "status";
471 + if (command === "on") {
472 + record(runtime, { op: "enabled", enabled: true, at: Date.now() });
473 + return "cmprss: on";
475 + if (command === "off") {
476 + record(runtime, { op: "enabled", enabled: false, at: Date.now() });
477 + return "cmprss: off";
479 + if (command === "clear") {
480 + record(runtime, { op: "clear", at: Date.now() });
481 + return "cmprss: cleared";
483 + if (command === "stats" || command === "status") {
484 + const saved = Math.max(
485 + 0,
486 + Math.floor(
487 + (runtime.stats.originalBytes - runtime.stats.compressedBytes) / 4,
488 + ),
489 + );
490 + return `cmprss: ${runtime.enabled ? "on" : "off"}; blocks=${runtime.stats.blocks}; saved~${saved}t; threshold=${runtime.config.thresholdBytes}B`;
492 + return "cmprss: usage: /cmprss [status|stats|on|off|clear]";
495 +function updateStatus(
496 + runtime: Runtime,
497 + ctx: {
498 + hasUI?: boolean;
499 + ui?: { setStatus?: (key: string, text: string | undefined) => void };
500 + },
501 +): void {
502 + if (!ctx.hasUI || !runtime.config.status) return;
503 + if (!runtime.enabled) {
504 + ctx.ui?.setStatus?.(STATUS_KEY, "cmprss: off");
505 + return;
507 + const saved = Math.max(
508 + 0,
509 + Math.floor(
510 + (runtime.stats.originalBytes - runtime.stats.compressedBytes) / 4,
511 + ),
512 + );
513 + ctx.ui?.setStatus?.(STATUS_KEY, `cmprss: ${saved}t`);
516 +export default function cmprssExtension(pi: ExtensionAPI) {
517 + let runtime = createRuntime(settingsFrom(pi), (event) =>
518 + pi.appendEntry(ENTRY_TYPE, event),
519 + );
521 + pi.on("session_start", (_event, ctx) => {
522 + runtime = createRuntime(settingsFrom(pi), (event) =>
523 + pi.appendEntry(ENTRY_TYPE, event),
524 + );
525 + replayState(
526 + runtime,
527 + ctx.sessionManager.getBranch() as Array<{
528 + type?: unknown;
529 + customType?: unknown;
530 + data?: unknown;
531 + }>,
532 + );
533 + updateStatus(runtime, ctx);
534 + });
536 + pi.on("session_shutdown", () => {
537 + runtime.blocks.clear();
538 + });
540 + pi.on("tool_result", ((event: any, ctx: any) => {
541 + const result = compressToolResult(runtime, event);
542 + updateStatus(runtime, ctx);
543 + return result;
544 + }) as any);
546 + pi.registerTool({
547 + name: "cmprss_retrieve",
548 + label: "cmprss retrieve",
549 + description: "Retrieve exact original text for a cmprss hash.",
550 + parameters: Type.Object({
551 + hash: Type.String({
552 + description: "cmprss hash from the compressed marker",
553 + }),
554 + }),
555 + async execute(_toolCallId, params) {
556 + const text = runtime.retrieve(params.hash);
557 + return {
558 + content: [{ type: "text", text }],
559 + details: { hash: params.hash, cmprssRetrieve: true },
560 + };
561 + },
562 + });
564 + pi.registerCommand("cmprss", {
565 + description: "Show or change cmprss state.",
566 + async handler(args, ctx) {
567 + const message = runCommand(runtime, args);
568 + updateStatus(runtime, ctx);
569 + if (ctx.hasUI) ctx.ui.notify(message, "info");
570 + },
571 + });
574 +export const __test = {
575 + DEFAULT_CONFIG,
576 + compressToolResult,
577 + createRuntime,
578 + replayState,
579 + runCommand,
580 + compressContent,

extensions/cmprss/package.json

Added
new file mode 100644
1 +{
2 + "name": "@bugabinga/pi-ext-cmprss",
3 + "type": "module",
4 + "main": "index.ts",
5 + "scripts": {},
6 + "devDependencies": {
7 + "@marcfargas/pi-test-harness": "^0.6.1",
8 + "vitest": "^4.1.9"
9 + },
10 + "peerDependencies": {
11 + "@earendil-works/pi-coding-agent": "*",
12 + "typebox": "*"
13 + },
14 + "license": "MIT",
15 + "description": "Persists and compresses oversized Pi tool results with exact retrieval.",
16 + "keywords": [
17 + "pi",
18 + "pi-extension"
19 + ],
20 + "private": true

extensions/ultra/__tests__/ui.test.ts

Modified
198 198 { expanded: false, isPartial: false },
199 199 { expanded: true, isPartial: true }, // expanded also reads `state.rows`
200 200 ] as const) {
201 - const comp = renderUltraResult({ content: [], details: foreign }, options, theme);
201 + const comp = renderUltraResult(
202 + { content: [], details: foreign },
203 + options,
204 + theme,
205 + );
202 206 expect(() => comp.render(80)).not.toThrow();
203 207 expect(comp.render(80).join("\n")).toContain("0/0 done");

extensions/ultra/ui.ts

Modified
170 170 theme: Theme,
171 171 _context?: unknown,
172 172 ): Component {
173 - return new UltraResultComponent(asReducerState(result?.details), options, theme);
173 + return new UltraResultComponent(
174 + asReducerState(result?.details),
175 + options,
176 + theme,
177 + );
176 180 // ---------------------------------------------------------------------------

package-lock.json

Modified
42 42 "@earendil-works/pi-coding-agent": "*"
45 + "extensions/cmprss": {
46 + "name": "@bugabinga/pi-ext-cmprss",
47 + "license": "MIT",
48 + "devDependencies": {
49 + "@marcfargas/pi-test-harness": "^0.6.1",
50 + "vitest": "^4.1.9"
51 + },
52 + "peerDependencies": {
53 + "@earendil-works/pi-coding-agent": "*",
54 + "typebox": "*"
55 + }
56 + },
45 57 "extensions/context-size": {
46 58 "name": "@bugabinga/pi-ext-context-size",
47 59 "license": "MIT",
1261 1273 "resolved": "extensions/builtin-tools",
1262 1274 "link": true
1276 + "node_modules/@bugabinga/pi-ext-cmprss": {
1277 + "resolved": "extensions/cmprss",
1278 + "link": true
1279 + },
1264 1280 "node_modules/@bugabinga/pi-ext-context-size": {
1265 1281 "resolved": "extensions/context-size",
1266 1282 "link": true