repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by bugabinga

Refine UI extension prototype findings

bd1c2c25a1b8e1d77d73abfc333adbf54d6aea16

Oliver Krylow <o.krylow@isp-insoft.de> ·

unsigned

Browse filesCompare with HEAD

Changed files (2)

docs/super/roadmaps/2026-07-01-ui-extension-roadmap.md

Modified
2 2 # UI Extension Roadmap
4 4 > **For agentic workers:** Use /skill:writing-plans to create one detailed implementation plan per phase. Start with Phase 1 and proceed sequentially unless the user explicitly changes the order.
6 -**Goal:** Build `/ui`, a persistent browser Pi frontend backed by a forked RPC session while terminal Pi remains independent.
6 +**Goal:** Build `/ui`, a persistent browser Pi frontend backed by an active-branch RPC session while terminal Pi remains independent.
8 8 **Design Spec:** [`docs/super/specs/2026-07-01-ui-extension-design.md`](../specs/2026-07-01-ui-extension-design.md)
27 27 **Out of scope:**
29 29 - RPC subprocess.
30 -- Forked session history.
30 +- Seeded active-branch session history.
31 31 - Full Pi web composer.
32 32 - Visual widgets and telemetry.
62 62 **Context notes:** Keep startup work inside `/ui`; extension load must stay cheap and side-effect-free.
64 -## Phase 2: Forked RPC Session and Basic Web Stream
64 +## Phase 2: Active-Branch RPC Session and Basic Web Stream
66 -**Outcome:** Browser UI talks to a forked persistent Pi RPC session and renders a basic ordered stream plus composer.
66 +**Outcome:** Browser UI talks to a persistent Pi RPC session seeded from the terminal active branch and renders a basic ordered stream plus composer.
68 68 **Why now:** This proves `/ui` is a real Pi frontend, not a companion page.
70 70 **Scope:**
72 -- Fork active terminal session branch into a persistent web session where possible.
73 -- Spawn `pi --mode rpc --session <fork>` with normal extensions/skills/settings.
72 +- Extract the active terminal session branch into a persistent web session with `createBranchedSession(getLeafId())` where possible.
73 +- Spawn `pi --mode rpc --session <seeded-session>` with normal extensions/skills/settings.
74 74 - Parse RPC stdout as LF-delimited JSONL and capture stderr separately.
75 75 - Implement server event bus, snapshot, replay buffer, WebSocket stream, and RPC stdin queue.
76 76 - Render basic browser shell: top state strip, central stream, tiny status strip, composer.
94 94 **Dependencies:**
96 96 - Phase 1.
97 -- Verified Pi session fork/RPC flags.
97 +- Verified Pi active-branch session extraction and RPC flags.
99 99 **Verification:**
101 -- Automated: fork path selected; parent recorded; JSONL parser splits LF only; event sequence monotonic; reconnect replays or snapshots; RPC death produces actionable event.
101 +- Automated: `createBranchedSession(getLeafId())` path selected; parent recorded; exact active branch preserved; whole-tree `forkFrom` fallback warns; JSONL parser splits LF only; event sequence monotonic; reconnect replays or snapshots; slow RPC startup and RPC death produce actionable events.
102 102 - Visual: open `/ui` and see seeded terminal history in the central stream with dominant `YOU` blocks.
103 103 - Visual: send a browser prompt and watch user message, assistant stream, and tool lifecycle appear in order.
104 104 - Visual: kill RPC process and confirm browser shows a clear, actionable stream error while terminal widget updates state.
108 108 **Risks:**
110 -- Exact active-branch fork APIs may differ; mitigate by failing with actionable error rather than ad-hoc corrupt JSON rewriting.
110 +- Exact active-branch extraction APIs may differ; mitigate by failing with actionable error rather than ad-hoc corrupt JSON rewriting.
111 111 - RPC event shapes may need normalization; mitigate with typed protocol tests before UI assumptions spread.
113 113 **Context notes:** Keep the stream deterministic and boring; defer polish until behavior is correct.
199 199 **Verification:**
201 -- Automated: stable widget ids; fallback text returned; mount/update/action/error events; `window.pi` method routing; widget errors become human+agent actionable stream errors.
201 +- Automated: stable widget ids; fallback text returned; mount/update/action/error events; `window.pi` method routing; Chrome/Firefox iframe bridge and bounded-layout probes pass; widget errors become human+agent actionable stream errors.
202 202 - Visual: ask agent for a visual card, form, diagram/prototype, and full HTML/CSS/JS widget; confirm all appear in the main stream, bounded and responsive.
203 203 - Visual: click widget controls and confirm resulting action is visibly attributed to that widget before reaching agent/session.
204 204 - Visual: trigger a widget JS error and confirm the block shows widget id/tool call id plus human action and agent fix hint.
209 209 **Risks:**
211 211 - Trusted full JS can leak data or impersonate actions; accepted by design, mitigated only through visible attribution, audit, and no silent widget-originated prompts.
212 -- CSS/layout interference can damage page usability; mitigate with document/layout boundary, containment, overflow, and explicit height negotiation.
212 +- CSS/layout interference can damage page usability; mitigate with document/layout boundary, containment, overflow, explicit height negotiation, and Chrome/Firefox mobile-width layout tests.
214 214 **Context notes:** Do not reintroduce sandbox planning unless user changes the threat model.

docs/super/specs/2026-07-01-ui-extension-design.md

Modified
24 24 - Web Pi session is independent and can run in parallel with terminal Pi.
25 25 - Web session is persistent and normal.
26 26 - Seed web session with full terminal active branch.
27 -- Use Pi fork semantics where possible.
27 +- Seed via exact active-branch extraction where possible; whole-tree fork is allowed only if explicitly chosen.
28 28 - Allow parallel edit races.
29 29 - Mobile support uses LAN HTTP + random token.
30 30 - Show QR code in terminal widget.
40 40 ```text
41 41 terminal pi session
42 42 └─ /ui command
43 - ├─ fork active terminal session branch into new persistent session
44 - ├─ spawn `pi --mode rpc --session <fork>` subprocess
43 + ├─ extract active terminal branch into new persistent session
44 + ├─ spawn `pi --mode rpc --session <seeded-session>` subprocess
45 45 ├─ start LAN HTTP/WebSocket server with random token
46 46 ├─ show terminal widget: OSC link, QR, ESC cancel, /ui off help
47 47 └─ keep terminal session independent
63 63 - token-authenticated browser app,
64 64 - WebSocket event channel,
65 65 - spawned Pi RPC subprocess,
66 -- forked persistent session file,
66 +- active-branch persistent session file,
67 67 - in-memory event store and replay buffer,
68 68 - telemetry aggregator,
69 69 - terminal TUI minimizer state.
87 87 ## Session seeding
89 -Use Pi session forking rather than ad-hoc JSON rewriting where possible.
89 +Use Pi `SessionManager` APIs rather than ad-hoc JSON rewriting.
91 -Preferred:
91 +Preferred exact active-branch path:
93 93 ```text
94 94 source = terminal ctx.sessionManager.getSessionFile()
95 -forked = SessionManager.forkFrom(source, ctx.cwd)
96 -spawn pi --mode rpc --session forked
95 +leaf = terminal ctx.sessionManager.getLeafId()
96 +seededPath = terminal ctx.sessionManager.createBranchedSession(leaf)
97 +spawn pi --mode rpc --session seededPath
99 -The fork must preserve the full active branch: user messages, assistant messages, thinking blocks, tool calls, tool results, bash execution messages, custom messages, compaction summaries, model changes, thinking-level changes, and relevant branch summaries.
100 +Prototype result: `SessionManager.forkFrom(source, cwd)` copies the whole session tree into the new file. That keeps LLM context clean because the active branch path is clean, but it also brings abandoned branches into `/tree`, increases history size, and may expose old branch history. Therefore use `createBranchedSession(leaf)` for the default exact active-branch seed. Whole-tree `forkFrom` is acceptable only behind an explicit option or fallback with visible warning.
102 +The seeded session must preserve the full active branch: user messages, assistant messages, thinking blocks, tool calls, tool results, bash execution messages, custom messages, compaction summaries, model changes, thinking-level changes, and relevant branch summaries.
101 104 The web session header records the terminal session as parent. The web session gets a clear display name such as `ui: <terminal session name or cwd>`.
109 112 Requirements:
111 114 - do not pass `--no-extension`, `--no-skills`, or `--no-session`,
112 -- pass the forked session via `--session`,
115 +- pass the seeded active-branch session via `--session`,
113 116 - inherit cwd,
114 117 - inherit normal environment,
115 118 - parse stdout strictly as JSONL split on LF only,
116 119 - capture stderr separately for diagnostics,
117 120 - restart only by explicit user action, not automatically,
118 -- expose RPC death as a stream error with fix instructions.
121 +- expose RPC startup, slow startup, and death as stream/terminal states with fix instructions.
123 +Prototype result: a normal RPC subprocess with full extensions can take longer than 8 seconds to answer first state requests. The browser and terminal widget must show an explicit “starting RPC” state instead of appearing frozen.
120 125 RPC commands used by the web UI:
491 496 - Mount each widget in a bounded visual block.
492 497 - Do not use iframe sandboxing or a separate permission wall for full JS.
493 498 - Use `iframe srcdoc`, Shadow DOM, or equivalent document/layout boundary only to keep CSS, layout, and lifecycle contained.
499 +- Prototype result: unsandboxed `iframe srcdoc` can call parent `window.pi` in system Chrome and Playwright Firefox.
500 +- Prototype result: a host-bounded iframe with intentionally over-wide/tall content can fit the stream on a 390px mobile viewport in system Chrome and Playwright Firefox when the host applies `width: 100%`, overflow containment, and capped height negotiation.
494 501 - Container gets fixed stream width, `max-width: 100%`, overflow boundary, and resize observer.
495 502 - Host applies a minimal reset and Nugu CSS variables.
496 503 - Widget document receives theme variables and container dimensions.
729 736 Session seed:
731 -- fork path selected when source session file exists,
732 -- full branch preserved,
738 +- `createBranchedSession(getLeafId())` selected when source session file exists,
739 +- exact active branch preserved,
740 +- whole-tree `forkFrom` fallback warns that abandoned branches are included,
733 741 - parent session recorded,
734 742 - display name set,
735 743 - ephemeral source fallback behaves as specified,
794 802 - `@` references and `!` bash parsing match Pi expectations,
795 803 - mobile layout hides nothing essential,
796 804 - iframe widget bridge exposes `window.pi`,
805 +- iframe widget layout does not create page-level horizontal overflow in Chrome or Firefox,
797 806 - widget height negotiation stays bounded.
799 808 ### E2E tests
805 814 - run `/ui`, obtain URL/token, open browser,
806 815 - QR URL and link route to same authenticated page,
807 -- initial snapshot shows forked session history,
816 +- initial snapshot shows exact active-branch seeded session history,
808 817 - send prompt from web, observe RPC events stream,
809 818 - invoke slash command,
810 819 - invoke `/skill:*`,
928 937 ## Open implementation risks
930 939 - Extension APIs cannot truly replace the whole terminal UI; design intentionally minimizes terminal UI instead.
931 -- Exact full-branch seeding may require `SessionManager.forkFrom` availability and behavior verification.
940 +- Exact active-branch seeding should use `createBranchedSession(getLeafId())`; `forkFrom` copies the whole tree and should not be the default.
932 941 - RPC mode does not support some interactive-only UI methods; web must emulate or degrade.
933 942 - `!`/`!!` exact terminal semantics may need explicit web-side mapping to RPC `bash`.
934 943 - Full JS widgets are powerful and risky; attribution/audit is mandatory.
941 950 Suggested phases:
943 952 1. Extension command + terminal widget + LAN auth server + QR.
944 -2. Forked RPC subprocess + event bridge + basic web stream/composer.
953 +2. Active-branch RPC subprocess + event bridge + basic web stream/composer.
945 954 3. Pi syntax, commands, skills, keybindings, status, extension UI protocol.
946 955 4. Visual tools and full-JS widget bridge.
947 956 5. Telemetry digest and lifecycle detection.