Luigit
repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by admin

extensions/canvas/DEBUGGING.md

Raw
Rendered preview

Debugging canvas

Session access

While drawing mode is active, the agent's canvas-mode prompt includes the full session URL. canvas({ action: "inspect" }) returns it under connection.url alongside the semantic scene and diagnostics. Opening canvas does not trigger an agent turn. Closing drawing mode removes the URL from active canvas-mode context and shuts down its server; reopening creates a new URL. Previously recorded prompts and tool results can retain expired URLs. The URL is an ephemeral access capability, not a persistent credential; keep it within the session rather than publishing it.

Use the full URL to open an inspection tab in Chrome or Firefox. If navigating from the same URL without its fragment, reload once so the application consumes the newly supplied fragment. The browser removes the fragment and keeps its token in tab-local session storage. Do not ask the human to retrieve it from their browser when canvas inspect is available. Opening an inspection tab does not reproduce another tab's unsaved edits, text-editing overlay, viewport, selection, or connection errors. Read shared state without editing; do not pan, change selection, or submit scene actions in the human's tab. Close inspection tabs when finished because each tab consumes a browser event-stream connection.

Runtime diagnostics

diagnostics reports live element count, in-progress generation (pendingDraw), connected editor streams (viewerCount), pending observation revision, retained changed-ID count, pending image presence (hasViewport), debounce state, humanEditing activity gate, agent-idle state, and pending-message state. After the layout helper is created, renderer reports launches, requests, running, and closed flags. A draw can succeed with viewerCount: 0; connected streams alone do not prove that a browser received or rendered a revision. The legacy viewport payload name now carries a full-scene export with scene bounds, not a viewport screenshot. inspect reports cached-image availability under image and attaches it to tool content only when its revision matches current state. observationsQueued counts accepted scene changes passed to the observation accumulator, not model turns. observationsDelivered counts successful calls to Pi's observation-message delivery API, not provider completion or visual comprehension. Both counters reset when drawing mode is newly activated and are not persisted. Inspecting from a model tool call normally reports agentIdle: false because that call belongs to an active run.

Accepted scenes save independently of activity ordering and image capture. Each browser reports monotonic activity sequences; stale activity cannot acknowledge or discard a scene save. Active pointer/text editing suspends delivery without discarding pending changes; finishing can deliver even if no document property changed on blur. Browser activity heartbeats renew a 30-second active lease and page exit reports disconnection. Already-settled work resumes when another editor disconnects or its lease expires; departure alone never marks an unfinished scene settled. The client activity map is bounded to eight entries and does not let an idle inspection tab override an active drawing tab. Image export runs after save acknowledgement on a cloned scene, then rechecks revision, edit epoch, and activity sequence before sending. A capture failure removes image availability for that revision rather than recycling older pixels.

Compare JSON and visible pixels from the same revision before attributing a failure to model understanding. Semantic inspection omits native drawing properties; it is not a lossless scene export. For native state, evaluate this read-only expression in the authorized inspection tab:

await fetch("/api/state", {
  headers: {
    Authorization: `Bearer ${sessionStorage.getItem("pi-canvas-token")}`,
  },
}).then((response) => response.json());

Inspect only needed fields; avoid dumping entire drawings or tokens into diagnostics. The UI's static canvas layer can omit text currently being edited in that tab. Agent imagery instead uses the public full-scene export API after interaction settles, not the editor's static canvas layer. Browser DOM text does not include text painted into the canvas.

Pi-owned generation and browser catch-up

createDraw produces a bounded immutable request, not a proposal for an editor tab. Pi generates a candidate, checks the current revision again, validates the candidate, persists the committed scene, and publishes it. The browser cannot prepare, commit, or cancel agent draws through /api/action; those obsolete actions are rejected. Direct moves, styles, and deletions use the plain document transform without launching Chromium. New elements, relabels, and Mermaid use the stateless /layout page in a canvas-owned temporary Chromium profile. That page has no editor, scene event stream, or authority to commit canonical state. Font loading uses the public native export path and local font assets before conversion, rather than measuring with fallback fonts. Abort, branch changes, and close cancel generation without publishing a partial candidate.

/api/changes?after=<revision> is authenticated and replays contiguous published revisions even when SSE notifications coalesce. The journal retains at most 64 snapshots within 8 MiB and pages responses within 1 MiB. An expired cursor receives the latest state with reset: true; a surviving clean tab applies that batch as one Undo step with a warning. Foreground visibility, focus, and page-show events trigger resync and refresh the event stream. The editor tracks its installed revision separately from the newest canonical revision, so an unapplied remote update never authorizes overwriting unseen edits. Each replayed agent revision gets a distinct native capture; remote human updates do not create agent Undo steps. Dirty drafts remain local and use the normal explicit conflict recovery actions. Reload starts a fresh native history, not a durable shared Undo log.

Native runtime and frozen-viewer proofs are documented in the source repository under extensions/canvas/__e2e__/README.md.

Opt-in event logs

Canvas uses the repository's extension debug contract. PI_CANVAS_DEBUG=1 enables file logging after Pi restarts; debugging remains off by default. Enabling logging creates files under the OS user debug directory, so obtain approval before enabling it during agent-driven investigations. Tests use isolated temporary log directories.

Safe events are declared in src/debug.ts: scene acceptance or no-op, observation queueing, deferral, delivery or failure, drawing request, commit or cancellation, and session lifecycle. Records include revisions, counts, viewport presence, and delivery-gate booleans, never URLs, tokens, drawing text, images, or raw errors. observation.deferred distinguishes debounce, active human editing, busy-agent, and pending-message gates. observation.queued reports both this scene change's ID count and the retained accumulated ID count, which is capped at 20. Logs do not prove provider receipt or successful interpretation; correlate them with session observations and model responses.

# Debugging canvas

## Session access

While drawing mode is active, the agent's canvas-mode prompt includes the full session URL.
`canvas({ action: "inspect" })` returns it under `connection.url` alongside the semantic scene and `diagnostics`.
Opening canvas does not trigger an agent turn.
Closing drawing mode removes the URL from active canvas-mode context and shuts down its server; reopening creates a new URL.
Previously recorded prompts and tool results can retain expired URLs.
The URL is an ephemeral access capability, not a persistent credential; keep it within the session rather than publishing it.

Use the full URL to open an inspection tab in Chrome or Firefox.
If navigating from the same URL without its fragment, reload once so the application consumes the newly supplied fragment.
The browser removes the fragment and keeps its token in tab-local session storage.
Do not ask the human to retrieve it from their browser when `canvas inspect` is available.
Opening an inspection tab does not reproduce another tab's unsaved edits, text-editing overlay, viewport, selection, or connection errors.
Read shared state without editing; do not pan, change selection, or submit scene actions in the human's tab.
Close inspection tabs when finished because each tab consumes a browser event-stream connection.

## Runtime diagnostics

`diagnostics` reports live element count, in-progress generation (`pendingDraw`), connected editor streams (`viewerCount`), pending observation revision, retained changed-ID count, pending image presence (`hasViewport`), debounce state, `humanEditing` activity gate, agent-idle state, and pending-message state.
After the layout helper is created, `renderer` reports launches, requests, running, and closed flags.
A draw can succeed with `viewerCount: 0`; connected streams alone do not prove that a browser received or rendered a revision.
The legacy `viewport` payload name now carries a full-scene export with scene bounds, not a viewport screenshot.
`inspect` reports cached-image availability under `image` and attaches it to tool content only when its revision matches current state.
`observationsQueued` counts accepted scene changes passed to the observation accumulator, not model turns.
`observationsDelivered` counts successful calls to Pi's observation-message delivery API, not provider completion or visual comprehension.
Both counters reset when drawing mode is newly activated and are not persisted.
Inspecting from a model tool call normally reports `agentIdle: false` because that call belongs to an active run.

Accepted scenes save independently of activity ordering and image capture.
Each browser reports monotonic activity sequences; stale activity cannot acknowledge or discard a scene save.
Active pointer/text editing suspends delivery without discarding pending changes; finishing can deliver even if no document property changed on blur.
Browser activity heartbeats renew a 30-second active lease and page exit reports disconnection.
Already-settled work resumes when another editor disconnects or its lease expires; departure alone never marks an unfinished scene settled.
The client activity map is bounded to eight entries and does not let an idle inspection tab override an active drawing tab.
Image export runs after save acknowledgement on a cloned scene, then rechecks revision, edit epoch, and activity sequence before sending.
A capture failure removes image availability for that revision rather than recycling older pixels.

Compare JSON and visible pixels from the same revision before attributing a failure to model understanding.
Semantic inspection omits native drawing properties; it is not a lossless scene export.
For native state, evaluate this read-only expression in the authorized inspection tab:

```js
await fetch("/api/state", {
  headers: {
    Authorization: `Bearer ${sessionStorage.getItem("pi-canvas-token")}`,
  },
}).then((response) => response.json());
```

Inspect only needed fields; avoid dumping entire drawings or tokens into diagnostics.
The UI's static canvas layer can omit text currently being edited in that tab.
Agent imagery instead uses the public full-scene export API after interaction settles, not the editor's static canvas layer.
Browser DOM text does not include text painted into the canvas.

## Pi-owned generation and browser catch-up

`createDraw` produces a bounded immutable request, not a proposal for an editor tab.
Pi generates a candidate, checks the current revision again, validates the candidate, persists the committed scene, and publishes it.
The browser cannot prepare, commit, or cancel agent draws through `/api/action`; those obsolete actions are rejected.
Direct moves, styles, and deletions use the plain document transform without launching Chromium.
New elements, relabels, and Mermaid use the stateless `/layout` page in a canvas-owned temporary Chromium profile.
That page has no editor, scene event stream, or authority to commit canonical state.
Font loading uses the public native export path and local font assets before conversion, rather than measuring with fallback fonts.
Abort, branch changes, and close cancel generation without publishing a partial candidate.

`/api/changes?after=<revision>` is authenticated and replays contiguous published revisions even when SSE notifications coalesce.
The journal retains at most 64 snapshots within 8 MiB and pages responses within 1 MiB.
An expired cursor receives the latest state with `reset: true`; a surviving clean tab applies that batch as one Undo step with a warning.
Foreground visibility, focus, and page-show events trigger resync and refresh the event stream.
The editor tracks its installed revision separately from the newest canonical revision, so an unapplied remote update never authorizes overwriting unseen edits.
Each replayed agent revision gets a distinct native capture; remote human updates do not create agent Undo steps.
Dirty drafts remain local and use the normal explicit conflict recovery actions.
Reload starts a fresh native history, not a durable shared Undo log.

Native runtime and frozen-viewer proofs are documented in the source repository under `extensions/canvas/__e2e__/README.md`.

## Opt-in event logs

Canvas uses the repository's [extension debug contract](../DEBUG.md).
`PI_CANVAS_DEBUG=1` enables file logging after Pi restarts; debugging remains off by default.
Enabling logging creates files under the OS user debug directory, so obtain approval before enabling it during agent-driven investigations.
Tests use isolated temporary log directories.

Safe events are declared in [`src/debug.ts`](src/debug.ts): scene acceptance or no-op, observation queueing, deferral, delivery or failure, drawing request, commit or cancellation, and session lifecycle.
Records include revisions, counts, viewport presence, and delivery-gate booleans, never URLs, tokens, drawing text, images, or raw errors.
`observation.deferred` distinguishes debounce, active human editing, busy-agent, and pending-message gates.
`observation.queued` reports both this scene change's ID count and the retained accumulated ID count, which is capped at 20.
Logs do not prove provider receipt or successful interpretation; correlate them with session observations and model responses.