repositories / dotfiles

dotfiles

bugabingas dorkfiles

owned by bugabinga

docs(neovim): plan pivi delivery from pi and neovim research

eeab014bcd8c0f491ec9e9e2c5ff56cc8c2f8e7c

Oliver Krylow <o.krylow@isp-insoft.de> ·

SSH signature · not checked

Browse filesCompare with HEAD

Changed files (3)

neovim/.system/plans/NL-PLAN-CEJPO_XF-deliver-pivi-native-neovim-integration/index.md

Added
new file mode 100644
1 +---
2 +id: NL-PLAN-CEJPO_XF
3 +type: plan
4 +title: Deliver pivi native Neovim integration
5 +spec: NL-SPEC-GMWDCMJZ
6 +status: approved
7 +---
9 +# Deliver pivi native Neovim integration
11 +## Intent
13 +Deliver [NL-SPEC-GMWDCMJZ](../../specs/NL-SPEC-GMWDCMJZ-pivi-native-neovim-integration/index.md) as vertical slices, each observable through Neovim surfaces.
14 +Evidence for transport, tool authority, and concurrency comes from [NL-RESEARCH-KZYYEFY_](../../research/NL-RESEARCH-KZYYEFY_-pi-and-neovim-capabilities-for-pivi/index.md).
16 +Placement: pivi is an ordinary configuration module at `lua/bugabinga/pivi/`, like the other Neovim modules, with its bundled Pi companion extension under `lua/bugabinga/pivi/extension/`.
17 +Module-level testing rules live in that directory's `AGENTS.md`.
18 +Publishing state through native variables and `User` events still holds, so other configuration consumes pivi without depending on its internals.
20 +## Module map
22 +```mermaid
23 +flowchart TB
24 + command["bugabinga.pivi.command"] --> session["bugabinga.pivi.session"]
25 + command --> help["bugabinga.pivi.help"]
26 + session --> rpc["bugabinga.pivi.rpc"]
27 + session --> status["bugabinga.pivi.status"]
28 + status --> publish["bugabinga.pivi.publish"]
29 + session --> transcript["bugabinga.pivi.transcript"]
30 + session --> context["bugabinga.pivi.context"]
31 + session --> dialog["bugabinga.pivi.dialog"]
32 + rpc --> extension["companion Pi extension"]
33 + extension --> edit["bugabinga.pivi.edit"]
34 + edit --> activity["bugabinga.pivi.activity"]
35 + activity --> follow["bugabinga.pivi.follow"]
36 + help --> rpc
37 +```
39 +| Module | Responsibility |
40 +| --- | --- |
41 +| `bugabinga.pivi.rpc` | Child process lifetime, strict LF JSONL framing, request correlation, deferral out of fast contexts |
42 +| `bugabinga.pivi.session` | Session identity, working directory, queue delivery, abort, resume, switch |
43 +| `bugabinga.pivi.status` | Cached semantic state; no RPC during status evaluation |
44 +| `bugabinga.pivi.publish` | `vim.g` state plus `User` autocmd events with payload data |
45 +| `bugabinga.pivi.transcript` | Prompt, transcript, and detail buffers |
46 +| `bugabinga.pivi.context` | Immutable submission snapshots and ambient editor state |
47 +| `bugabinga.pivi.edit` | Buffer-authoritative mutation, undo coherence, disk visibility |
48 +| `bugabinga.pivi.activity` | Extmarks, signs, virtual text, diagnostics, quickfix and location lists |
49 +| `bugabinga.pivi.follow` | Optional viewport following |
50 +| `bugabinga.pivi.dialog` | Extension UI requests mapped to `vim.ui` surfaces |
51 +| `bugabinga.pivi.help` | Ephemeral session-less run |
52 +| `bugabinga.pivi.command` | `:Pi` subcommands, slash-command entry, `<Plug>` mappings |
53 +| companion extension | Editor tools plus buffer-authoritative operations for built-in Pi tools |
55 +## Slice order
57 +```mermaid
58 +flowchart LR
59 + s1["1 Ask and stream"] --> s3["3 Status and events"]
60 + s1 --> s2["2 Ephemeral help"]
61 + s3 --> s4["4 Session control"]
62 + s4 --> s5["5 Submission context"]
63 + s5 --> s6["6 Buffer-authoritative reads"]
64 + s6 --> s7["7 Native edits"]
65 + s7 --> s8["8 Activity and lists"]
66 + s7 --> s9["9 Streaming ghost text"]
67 + s8 --> s10["10 Follow mode"]
68 + s4 --> s11["11 Slash commands"]
69 + s4 --> s12["12 Extension dialogs"]
70 +```
72 +| # | Observable behavior | Modules touched | Verified by |
73 +| --- | --- | --- | --- |
74 +| 1 | `:Pi ask {text}` starts Pi, streams assistant text into a transcript buffer, and survives partial reads | `bugabinga.pivi.rpc`, `bugabinga.pivi.session`, `bugabinga.pivi.transcript`, `bugabinga.pivi.command` | Framing and assembly specs plus a live round trip |
75 +| 2 | `:Pi help {q}` answers from an independent session-less run without touching any working session | `bugabinga.pivi.help`, `bugabinga.pivi.command` | Concurrent-run spec asserting untouched session history, queue, and status |
76 +| 3 | Status changes update cached state, emit semantic `User` events, and never issue RPC during evaluation | `bugabinga.pivi.status`, `bugabinga.pivi.publish` | Event payload spec plus a consumer that degrades when pivi is absent |
77 +| 4 | Create, resume, switch, stop sessions; queue steering or follow-up during a run; list, clear, abort | `bugabinga.pivi.session`, `bugabinga.pivi.command`, `bugabinga.pivi.transcript` | Queue state-machine spec matching the spec's run states |
78 +| 5 | Submissions carry immutable buffer, revision, cursor, range, and selection snapshots; queued items keep their own | `bugabinga.pivi.context`, `bugabinga.pivi.session` | Snapshot immutability spec under later buffer edits |
79 +| 6 | Pi reads unsaved loaded-buffer content and current core editor state on demand, including arbitrary Lua | companion extension, `bugabinga.pivi.rpc` | Tool-result spec comparing buffer versus disk content |
80 +| 7 | A normal Pi edit lands in a loaded buffer without reload, stays undoable as one change, and is visible to later shell work | `bugabinga.pivi.edit`, companion extension | Undo and coherence spec covering join failure fallback |
81 +| 8 | Edits leave a navigable trail; findings populate quickfix, location lists, and a distinct diagnostic namespace | `bugabinga.pivi.activity` | Namespace isolation and navigation spec |
82 +| 9 | Streaming tool arguments render ghost text that never mutates a buffer before arguments are complete | `bugabinga.pivi.activity`, `bugabinga.pivi.edit` | Partial-argument spec asserting zero mutation |
83 +| 10 | Follow mode tracks located events in a dedicated window or tab and restores the prior location | `bugabinga.pivi.follow` | Window ownership and restore spec |
84 +| 11 | `:Pi command ...` runs discovered Pi commands; `/` opens entry only at the first character of an input buffer | `bugabinga.pivi.command` | Cursor-position and discovery spec |
85 +| 12 | Extension dialogs and fire-and-forget status map to native surfaces and answer with matching identifiers | `bugabinga.pivi.dialog` | Request and response correlation spec |
87 +Slice 1 is the walking skeleton; every later slice keeps `just test` green before the next begins.
89 +## Critical paths
91 +- Child output arrives in fast contexts; all editor work defers through scheduling before API calls.
92 +- Submission during an active run always chooses steering or follow-up; rejection is never silent.
93 +- Help spawns its own session-less child so working-session history, queue, and status remain untouched.
94 +- Buffer authority prefers injecting operations into Pi's built-in tools over reimplementing them, because result and detail shapes must match exactly.
96 +## Risks
98 +| Risk | Slice | Mitigation |
99 +| --- | --- | --- |
100 +| Undo join refuses after undo or redo | 7 | Detect refusal and fall back to separate undo blocks rather than losing the edit |
101 +| Partial tool arguments are invalid JSON | 9 | Treat ghost text as presentation only; mutate solely on complete validated arguments |
102 +| Tool override breaks rendering or session state | 6, 7 | Keep built-in renderers and detail shapes; override behavior only |
103 +| Status consumers stall the statusline | 3 | Read cached state only |
104 +| Agent shell work reads stale disk content | 7 | Make agent buffer edits visible to filesystem readers as part of the edit behavior |

neovim/.system/research/NL-RESEARCH-KZYYEFY_-pi-and-neovim-capabilities-for-pivi/index.md

Added
new file mode 100644
1 +---
2 +id: NL-RESEARCH-KZYYEFY_
3 +type: research
4 +title: Pi and Neovim capabilities for pivi
5 +---
7 +# Pi and Neovim capabilities for pivi
9 +## Sources
11 +Installed documentation only; no remembered API behavior.
13 +| Source | Version | Location |
14 +| --- | --- | --- |
15 +| Pi docs (`rpc.md`, `extensions.md`, `usage.md`, `sessions.md`, `json.md`) | 0.85.1 | `$PI_CONTEXT_DOCS_PATH` |
16 +| Neovim runtime docs (`api.txt`, `lua.txt`, `undo.txt`, `editing.txt`, `options.txt`) | 0.12.5 | `$VIMRUNTIME/doc` |
18 +## Pi facts
20 +### Transport
22 +- `pi --mode rpc` speaks JSON commands on stdin, responses and events on stdout.
23 +- Strict JSONL, LF only; clients strip a trailing `\r`; Unicode separators must not split records.
24 +- Optional `id` on commands correlates responses; most events carry no `id`.
25 +- Startup flags relevant to a client: `--provider`, `--model`, `--name`, `--session <path|id>`, `--session-dir`, `--no-session`, `-e <ext>`, `--tools`, `--append-system-prompt`.
27 +### Commands that map to pivi surfaces
29 +| pivi need | RPC command | Notes |
30 +| --- | --- | --- |
31 +| Submit prompt | `prompt` | During streaming requires `streamingBehavior: steer \| followUp`, else error |
32 +| Queue while running | `steer`, `follow_up`, `clear_queue` | Extension commands rejected here |
33 +| Stop | `abort`, `abort_retry`, `abort_bash` | `abort` waits for idle |
34 +| Status | `get_state`, `get_session_stats` | Model, thinking, streaming, queue, context usage |
35 +| Transcript | `get_messages`, `get_entries`, `get_tree` | `get_entries since` is a durable cursor; `leafId` detects branch moves |
36 +| Sessions | `new_session`, `switch_session`, `fork`, `clone`, `set_session_name` | Switch/fork cancellable by extensions |
37 +| Slash commands | `get_commands` | Returns extension, prompt-template, and skill commands; invoked as `/name` through `prompt` |
38 +| Model/thinking | `set_model`, `cycle_model`, `get_available_models`, `set_thinking_level`, `get_available_thinking_levels` | |
39 +| Compaction | `compact`, `set_auto_compaction` | |
41 +Built-in TUI commands such as `/settings` and `/hotkeys` are absent from `get_commands` and do not execute over RPC.
43 +### Concurrency semantics
45 +```mermaid
46 +stateDiagram-v2
47 + [*] --> Idle
48 + Idle --> Streaming: prompt
49 + Streaming --> Streaming: steer / follow_up queued
50 + Streaming --> Streaming: extension command executes immediately
51 + Streaming --> Idle: agent_settled
52 + Streaming --> Idle: abort
53 +```
55 +- A plain `prompt` while streaming is rejected without `streamingBehavior`.
56 +- An extension command (`/name`) sent through `prompt` executes immediately even during streaming and manages its own LLM interaction.
57 +- `agent_end` marks one low-level run; `agent_settled` marks full settlement after retry, compaction, and queued continuations.
58 +- Therefore a truly ephemeral question that never enters session history needs either an extension command or a separate process (`pi -p` / `pi --mode json` / a second `--mode rpc --no-session` child).
60 +### Events
62 +Streamed as JSONL: `agent_start`, `agent_end`, `agent_settled`, `turn_start`, `turn_end`, `message_start`, `message_update`, `message_end`, `tool_execution_start|update|end`, `bash_execution_update`, `queue_update`, `compaction_start|end`, `auto_retry_start|end`, `summarization_retry_*`, `extension_error`.
64 +- `message_update` carries only deltas; clients assemble partial state by `contentIndex`; `message_end.message` is authoritative.
65 +- Tool-call streaming exposes `toolcall_start` (`id`, `toolName`), `toolcall_delta` (argument chunks), `toolcall_end` (complete call). This is the only source for progressive edit intent, and argument chunks are partial JSON text.
66 +- `tool_execution_update.partialResult` is cumulative, not a delta.
68 +### Extension UI over RPC
70 +| Method | RPC behavior |
71 +| --- | --- |
72 +| `select`, `confirm`, `input`, `editor` | Request/response sub-protocol with matching `id`; optional `timeout` auto-resolves agent-side |
73 +| `notify`, `setStatus`, `setWidget`, `setTitle`, `set_editor_text` | Fire-and-forget requests |
74 +| `custom` | Returns `undefined` |
75 +| `setWorkingMessage`, `setWorkingIndicator`, `setFooter`, `setHeader`, `setEditorComponent`, `setToolsExpanded` | No-ops |
76 +| `getEditorText`, `getToolsExpanded`, theme getters/setters | Degraded or empty |
78 +`ctx.mode === "rpc"` and `ctx.hasUI === true`; TUI-only capability must be guarded on `ctx.mode === "tui"`.
80 +### Extension hooks a companion extension can use
82 +- `pi.registerTool` at load or at runtime; new tools are callable without `/reload`.
83 +- Built-in tools `read`, `bash`, `powershell`, `edit`, `write`, `grep`, `find`, `ls` can be overridden by registering the same name; result and `details` shapes must match exactly; omitted renderers fall back to built-ins; `promptSnippet`/`promptGuidelines` are not inherited.
84 +- Built-in tools also accept pluggable operations (`ReadOperations`, `WriteOperations`, `EditOperations`, `BashOperations`, …) via `createReadTool` and siblings, plus a `spawnHook` for shell tools. This delegates I/O without reimplementing tool semantics.
85 +- `tool_call` can mutate arguments or block; `tool_result` can post-process.
86 +- `input` event sees raw text with `source` of `interactive | rpc | extension` and can `continue`, `transform`, or `handled`.
87 +- `before_agent_start` exposes and can extend `systemPrompt` and structured `systemPromptOptions`.
88 +- `pi.registerCommand` supports `getArgumentCompletions`; `pi.getCommands` mirrors `get_commands`.
89 +- Tool output must be truncated; built-in limits are 50KB and 2000 lines.
91 +## Neovim facts
93 +### Transport
95 +- `vim.system(cmd, { stdin = true, stdout = fun(err, data), stderr = fun(err, data) })` gives streaming callbacks plus `SystemObj:write(data)` and `obj:write(nil)` to close stdin; `obj:kill()` stops the child.
96 +- Output callbacks run in fast contexts; API calls must be deferred through `vim.schedule` (`api-fast`, `vim.in_fast_event()`).
97 +- `jobstart`/`nvim_chan_send` remain the alternative channel API.
99 +### Buffers and edits
101 +| Need | Mechanism | Caveat |
102 +| --- | --- | --- |
103 +| Precise edit | `nvim_buf_set_text`, `nvim_buf_set_lines` | Each call opens a new undo block |
104 +| One undoable change | `:undojoin` before the next change | `E790` after undo/redo; must not be used blindly |
105 +| Change detection | `nvim_buf_attach` `on_lines`, `b:changedtick` | Callbacks are fast-context |
106 +| Diffing model output against buffer | `vim.text.diff` with `result_type = 'indices'` | Available in 0.12 |
107 +| Disk coherence after buffer edits | buffer write plus `:checktime`; `'autoread'` defaults on | Unwritten buffers stay invisible to filesystem tools |
109 +### Presentation
111 +- Extmarks provide `virt_text` with `virt_text_pos = "inline"` for ghost text, `virt_lines` (with `virt_lines_above`, `virt_lines_overflow`), highlights, and signs.
112 +- `nvim_set_decoration_provider` with `ephemeral` marks renders per redraw without persistent state.
113 +- `vim.diagnostic` namespaces keep agent findings separate from LSP and compiler diagnostics.
114 +- Quickfix and location lists use `setqflist`/`setloclist` and normal navigation.
116 +### Integration surface
118 +- `nvim_exec_autocmds` emits `User` events with a `data` payload (`event-data`), which fits publishing cached pivi state.
119 +- `nvim_create_user_command` supplies subcommands, `nargs`, and custom completion for `:Pi help` and `:Pi command ...`.
120 +- `<Plug>` mappings expose behavior without claiming global keys.
121 +- `vim.ui.select` and `vim.ui.input` can back RPC dialog methods.
123 +## Spec claims versus evidence
125 +| Spec claim | Support | Risk |
126 +| --- | --- | --- |
127 +| Non-terminal JSONL subprocess channel | `pi --mode rpc` plus `vim.system` stdin/stdout | Low |
128 +| Semantic status (queue, run, tool, retry, compaction) | `get_state`, `get_session_stats`, event stream | Low |
129 +| Slash commands via `:Pi command ...` | `get_commands` plus `prompt` with `/name` | Low; TUI-only commands unavailable |
130 +| `:Pi help` runnable during an active run | Extension command executes during streaming, or a separate ephemeral process | Medium; not ephemeral if routed through the working session |
131 +| Loaded buffers authoritative for reads and edits | Built-in tool override or injected `ReadOperations`/`EditOperations`/`WriteOperations` | Medium; result and `details` shapes must match exactly |
132 +| Progressive ghost text from streaming tool arguments | `toolcall_delta` plus inline `virt_text` | Medium; requires tolerant partial-JSON parsing |
133 +| One coherent undoable change per edit | `nvim_buf_set_text` plus `:undojoin` | Medium; `E790` and multi-hunk edits |
134 +| Edits visible to later shell work without watchers | Explicit buffer write plus `:checktime` | Medium; unwritten buffers diverge from disk |
135 +| Extension dialogs mapped to native surfaces | `extension_ui_request`/`extension_ui_response` | Low |
136 +| Multiple sessions, resume, switch | `--session`, `switch_session`, `new_session`, `fork`, `clone` | Low |
138 +## Open questions
140 +- Does `:Pi help` use a second short-lived Pi process, or a companion extension command inside the working session that suppresses history?
141 +- Can two Pi processes safely target one session file, or must ephemeral help always run with `--no-session`?
142 +- Is buffer authority better implemented as operations injection into built-in tools than as full tool overrides, given the exact `details` shape requirement?
143 +- How should partial `toolcall_delta` arguments be parsed safely enough to show ghost text without ever mutating a buffer?
144 +- When must pivi write a modified buffer to disk so that agent shell commands observe agent edits?

neovim/.system/specs/NL-SPEC-GMWDCMJZ-pivi-native-neovim-integration/index.md

Modified
3 3 id: NL-SPEC-GMWDCMJZ
4 4 type: spec
5 5 title: pivi Native Neovim Integration
6 +research:
7 + - NL-RESEARCH-KZYYEFY_
8 10 # pivi Native Neovim Integration
35 37 - A tab may select a default session but does not own its lifetime.
36 38 - Source buffers do not own sessions.
37 39 - Closing a session window hides presentation without deleting history or implicitly stopping Pi.
40 +- While a session run is active, a new submission is queued with an explicit delivery choice and is never silently dropped.
41 +- Queued submissions remain listable and clearable before delivery.
43 +```mermaid
44 +stateDiagram-v2
45 + [*] --> Idle
46 + Idle --> Running: submit
47 + Running --> Running: queue steering or follow-up
48 + Running --> Idle: settle
49 + Running --> Idle: abort
50 + Idle --> [*]: stop session
51 +```
39 53 ## Slash commands
41 55 - Pi slash commands are available through `:Pi command ...`.
56 +- Only commands Pi exposes to non-interactive clients are offered; terminal-only commands are absent.
42 57 - In a Pi input buffer, `/` invokes slash-command entry only when the cursor is at the first character.
43 58 - Elsewhere, `/` retains ordinary Neovim behavior.
45 60 ## One-shot help
47 -- `:Pi help` asks a fast, ephemeral question about Neovim or pivi.
48 -- Help runs independently of persistent Pi sessions and does not enter their history.
49 -- Help remains available while another agent run is active and neither queues behind nor interrupts that run.
62 +- `:Pi help` answers a fast question about Neovim or pivi.
63 +- Help runs as an independent ephemeral agent run with no persisted session and no shared history.
64 +- Help never enters a working session's transcript, queue, or status.
65 +- Help stays available while a working session runs, and neither delays nor interrupts it.
66 +- Help failure is isolated and never degrades a working session.
68 +```mermaid
69 +flowchart LR
70 + editor["Neovim"] --> session["Working session run"]
71 + editor --> help["Ephemeral help run"]
72 + session --> history["Persistent session history"]
73 + help --> discard["Discarded after answer"]
74 +```
51 76 ## Editor context
122 147 ## Acceptance criteria
124 149 - A Pi slash command can run through `:Pi command ...`; `/` invokes slash-command entry only at the first character of a Pi input buffer.
125 -- `:Pi help` can answer an ephemeral Neovim or pivi question while another agent run remains active and unaffected.
150 +- `:Pi help` answers while a working session run is active, without appearing in that session's history, queue, or status.
151 +- A submission made during an active run is queued with explicit delivery and can be listed and cleared before delivery.
126 152 - A user can create, resume, switch, and stop explicit Pi sessions without tying their lifetime to a source buffer or tab.
127 153 - A prompt can include an unsaved selection with stable source and revision metadata.
128 154 - Pi can read unsaved loaded-buffer content and use current core Neovim context on demand.