repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by admin

feat(firefox-bidi): promote prototype extension from firefox-bidi repo

f8310b344edc70329a9a33fd03a9519009bbafb4

Oliver Jan Krylow <oliver@bugabinga.net> ·

SSH signature · not checked

Browse filesCompare with HEAD

Raw WebDriver BiDi tool: batch of {id?, method, params} frames, verbatim
responses plus captured events, {{id.result.path}} interpolation across
batches, oversized-result spooling with deferred drain, and one managed
headless Firefox per session with fresh temp profiles.

Contract adaptations over the prototype:
- tool registered dormant; session_start removes it from the active set
  until /firefox (or the PI_FIREFOX=1 dev hook) activates it
- bundled skill contributed via resources_discover instead of .pi/skills
- console diagnostics replaced by opt-in file log (PI_FIREFOX_DEBUG)
- /firefox argument autocomplete for close
- FIREFOX_BIN/PATH binary resolution with win32 handling, load-time
  warning, and fast not-found failure
- vitest + pi-test-harness coverage: real-runtime load, dormant/activate/
  close lifecycle, missing-binary and fake-firefox paths; lib unit tests
- bench tasks for the batch hot path (micro + real Firefox e2e)

Changed files (14)

extensions/firefox-bidi/README.md

Added
new file mode 100644
1 +# firefox-bidi
3 +Drive Firefox from Pi with raw W3C WebDriver BiDi frames: send a batch of `{id?, method, params}` frames, receive verbatim responses plus captured events.
5 +Protocol knowledge, launch policy, and Firefox capability deltas live in the bundled skill: `skills/firefox-bidi/SKILL.md`.
7 +## Usage
9 +- `/firefox` — activate the `firefox_bidi` tool; Firefox launches lazily on first call with a fresh temp profile.
10 +- `/firefox close` — shut the browser down and deactivate the tool.
11 +- `firefox_bidi` — batch raw BiDi frames; `{{id.result.path}}` placeholders reference earlier responses in the same batch or a previous one; oversized results spool to disk.
13 +## Requirements
15 +- Firefox on `PATH`, or `FIREFOX_BIN` pointing at the binary.
16 +- Missing binaries produce a startup warning and a fast, clear tool error.
18 +## Settings
20 +`firefox-bidi.screencastDir` (project or user `settings.json`) sets the default `browsingContext.startScreencast` output directory.
22 +## Benchmarks
24 +- `mise run //extensions/firefox-bidi:bench` — batch hot path, fake socket.
25 +- `mise run //extensions/firefox-bidi:bench-e2e` — same batch shape against a real headless Firefox.
26 +- Methodology: `__bench__/README.md`.

extensions/firefox-bidi/__bench__/README.md

Added
new file mode 100644
1 +# Benchmark: batch hot path
3 +## Scope and exclusions
5 +In-process hot path of the `firefox_bidi` tool: per-frame processing
6 +(`runBatch`) and payload serialization, plus its interaction with real
7 +Firefox round-trips. Excluded: browser launch and model latency — they
8 +dominate end-to-end wall time and are not this repo's code.
10 +## Measurements
12 +Two matched benchmarks:
14 +1. **Microbenchmark** (`__bench__/runBatch.bench.mts`): fake socket, fixture
15 + batch of 25 frames (22 small + 3 × 512 KB results). Isolates CPU cost
16 + (interpolation, serialization, ledger, spool writes) from the network.
17 +2. **E2E benchmark** (`__bench__/e2e.mts`): real headless Firefox, one
18 + `/session` socket, real round-trips; 5-frame batches with three real
19 + 300 KB `script.evaluate` results that hit the spool path.
21 +Both: interleaved A/B pairs (drift cancellation), fixture/launch setup
22 +outside the timer, output-equality gates, Wilcoxon signed-rank on paired
23 +differences (normal approximation).
25 +## Findings
27 +Microbenchmark (before optimization): `JSON.stringify` of a 512 KB response
28 +≈ 0.11 ms; the old loop serialized every response twice (size check + full
29 +payload re-stringify) ≈ 0.36 ms/batch; spool `writeFile` calls sat on the
30 +frame critical path. n=30 paired: no significant difference (p=0.69) — fs
31 +writes dominated the loop.
33 +E2E (the decisive measurement): spool writes deferred to a post-loop drain
34 +(`Promise.all`) so they overlap subsequent frame round-trips:
36 +| run | inline-spool median | deferred-spool median | paired diff | p |
37 +|---|---|---|---|---|
38 +| 12 pairs | 55.2 ms | 27.3 ms | 26.7 ms | 0.008 |
39 +| 15 pairs | 53.7 ms | 25.3 ms | 24.5 ms | 0.0022 |
41 +≈ **46% faster batches** containing oversized payloads. Additional micro
42 +optimizations kept: single serialization per response (size check shares the
43 +payload text), `interpolate` fast-path for strings without `{{`.
45 +## Decision
47 +Deferred spool drain is behavior-preserving (files are flushed before the
48 +tool returns; write errors still surface) and measured significant against a
49 +real browser. Serialization and interpolation changes are measured neutral
50 +to positive and strictly reduce work.
52 +## Reproduce
54 +```sh
55 +mise run //extensions/firefox-bidi:bench
56 +mise run //extensions/firefox-bidi:bench-e2e -- 15
57 +```

extensions/firefox-bidi/__bench__/baseline.ts

Added
new file mode 100644
1 +// Verbatim copy of the pre-optimization hot path (index.ts runBatch loop +
2 +// execute payload build), used as the benchmark baseline and diff-checked
3 +// against the optimized implementation. Do not "improve" this file.
5 +import { mkdtemp, writeFile } from "node:fs/promises";
6 +import { tmpdir } from "node:os";
7 +import { join } from "node:path";
8 +import { interpolate } from "../lib.ts";
10 +const MAX_INLINE = 256 * 1024;
12 +interface BidiResponse {
13 + id: number;
14 + method?: string;
15 + type?: string;
16 + result?: unknown;
17 + error?: string;
18 + message?: string;
19 + skipped?: boolean;
22 +interface BatchCtx {
23 + pending: Map<number, (r: BidiResponse) => void>;
24 + ledger: {
25 + subscriptions: string[];
26 + intercepts: string[];
27 + preloadScripts: string[];
28 + };
29 + spoolDir: string;
32 +function ledgerUpdate(
33 + b: BatchCtx,
34 + method: string,
35 + params: Record<string, unknown>,
36 + res: BidiResponse,
37 +): void {
38 + const ok = res.type === "success";
39 + const result = (res.result ?? {}) as Record<string, unknown>;
40 + if (method === "session.subscribe" && ok) {
41 + for (const e of (params.events as string[]) ?? [])
42 + if (!b.ledger.subscriptions.includes(e)) b.ledger.subscriptions.push(e);
43 + } else if (method === "session.unsubscribe") {
44 + const gone = new Set((params.events as string[]) ?? []);
45 + b.ledger.subscriptions = b.ledger.subscriptions.filter((x) => !gone.has(x));
46 + } else if (
47 + method === "network.addIntercept" &&
48 + ok &&
49 + typeof result.intercept === "string"
50 + ) {
51 + b.ledger.intercepts.push(result.intercept);
52 + } else if (method === "network.removeIntercept") {
53 + const ids = new Set(
54 + ((params.intercepts as string[]) ?? []).concat(
55 + (params.intercept as string[]) ?? [],
56 + ),
57 + );
58 + b.ledger.intercepts = b.ledger.intercepts.filter((x) => !ids.has(x));
59 + } else if (
60 + method === "script.addPreloadScript" &&
61 + ok &&
62 + typeof result.id === "string"
63 + ) {
64 + b.ledger.preloadScripts.push(result.id);
65 + } else if (
66 + method === "script.removePreloadScript" &&
67 + typeof params.id === "string"
68 + ) {
69 + b.ledger.preloadScripts = b.ledger.preloadScripts.filter(
70 + (x) => x !== params.id,
71 + );
72 + }
75 +export async function runBatchBaseline(
76 + ws: { send: (s: string) => void },
77 + b: BatchCtx,
78 + frames: { id: number; method: string; params?: Record<string, unknown> }[],
79 + byId: Map<number, BidiResponse>,
80 + timeoutMs: number,
81 +): Promise<BidiResponse[]> {
82 + const responses: BidiResponse[] = [];
83 + let aborted = false;
84 + for (const frame of frames) {
85 + if (aborted) {
86 + responses.push({ id: frame.id, method: frame.method, skipped: true });
87 + continue;
88 + }
89 + let wire: unknown;
90 + try {
91 + wire = {
92 + id: frame.id,
93 + method: frame.method,
94 + params: interpolate(frame.params ?? {}, byId),
95 + };
96 + } catch (e) {
97 + responses.push({
98 + id: frame.id,
99 + method: frame.method,
100 + type: "error",
101 + error: "pi interpolation",
102 + message: e instanceof Error ? e.message : String(e),
103 + });
104 + aborted = true;
105 + continue;
107 + const res = await new Promise<BidiResponse>((resolve) => {
108 + const t = setTimeout(
109 + () =>
110 + resolve({
111 + id: frame.id,
112 + type: "timeout",
113 + message: `no response within ${timeoutMs}ms; side effects may have applied`,
114 + }),
115 + timeoutMs,
116 + );
117 + b.pending.set(frame.id, (r) => {
118 + clearTimeout(t);
119 + resolve(r);
120 + });
121 + ws.send(JSON.stringify(wire));
122 + });
123 + ledgerUpdate(
124 + b,
125 + frame.method,
126 + (frame.params ?? {}) as Record<string, unknown>,
127 + res,
128 + );
129 + byId.set(frame.id, res);
130 + const out: BidiResponse = { ...res, id: frame.id, method: frame.method };
131 + const size = JSON.stringify(out).length;
132 + if (size > MAX_INLINE && res.type === "success") {
133 + const file = join(
134 + b.spoolDir,
135 + `${frame.method.replace(/[^\w.-]/g, "_")}-${frame.id}.json`,
136 + );
137 + await writeFile(file, JSON.stringify(out.result ?? null));
138 + out.result = {
139 + spooled: file,
140 + size,
141 + note: "full result written to file; read selectively",
142 + } as unknown;
144 + responses.push(out);
146 + return responses;
149 +export function payloadBaseline(
150 + session: unknown,
151 + responses: BidiResponse[],
152 + events: unknown[],
153 + wait: unknown,
154 + eventCap: number,
155 +): string {
156 + const payload = {
157 + session,
158 + responses,
159 + events,
160 + ...(events.length === eventCap
161 + ? { note: `event buffer capped at ${eventCap}` }
162 + : {}),
163 + ...(wait ? { wait } : {}),
164 + };
165 + return JSON.stringify(payload);
168 +// ── fixture ───────────────────────────────────────────────────────────────
170 +export interface Fixture {
171 + ws: { send: (s: string) => void };
172 + ctx: BatchCtx;
173 + frames: { id: number; method: string; params?: Record<string, unknown> }[];
174 + byId: Map<number, BidiResponse>;
177 +// Batch shape a real agent call produces: many small frames, a few huge
178 +// (screenshot/print-like) results, some params with {{N.result.*}} refs.
179 +export async function makeFixture(
180 + _smallKb = 2,
181 + bigKb = 512,
182 + small = 22,
183 + big = 3,
184 +): Promise<Fixture> {
185 + const pending = new Map<number, (r: BidiResponse) => void>();
186 + const frames: Fixture["frames"] = [];
187 + const resultsById = new Map<number, unknown>();
188 + let id = 1;
189 + frames.push({
190 + id: id,
191 + method: "session.subscribe",
192 + params: { events: ["network"] },
193 + });
194 + resultsById.set(id, {});
195 + id++;
196 + frames.push({ id: id, method: "browsingContext.create", params: {} });
197 + const ctxId = `ctx-${id}-aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee`;
198 + resultsById.set(id, { context: ctxId, userContext: "default" });
199 + id++;
200 + frames.push({
201 + id: id,
202 + method: "browsingContext.navigate",
203 + params: {
204 + context: `{{2.result.context}}`,
205 + url: "https://example.com/",
206 + wait: "complete",
207 + },
208 + });
209 + resultsById.set(id, { context: ctxId, url: "https://example.com/" });
210 + id++;
211 + for (let i = 0; i < small; i++) {
212 + frames.push({
213 + id: id,
214 + method: "script.evaluate",
215 + params: {
216 + expression: `1+${i}`,
217 + awaitPromise: false,
218 + target: { context: "{{2.result.context}}" },
219 + },
220 + });
221 + resultsById.set(id, { type: "number", value: i });
222 + id++;
224 + for (let i = 0; i < big; i++) {
225 + frames.push({
226 + id: id,
227 + method: "browsingContext.captureScreenshot",
228 + params: { context: "{{2.result.context}}" },
229 + });
230 + resultsById.set(id, {
231 + data: "iVBORw0KGgoAAAANSUhEUg".padEnd(bigKb * 1024, "A"),
232 + });
233 + id++;
235 + const ws = {
236 + send: (s: string) => {
237 + const f = JSON.parse(s);
238 + queueMicrotask(() =>
239 + pending.get(f.id)?.({
240 + id: f.id,
241 + type: "success",
242 + result: resultsById.get(f.id),
243 + }),
244 + );
245 + },
246 + };
247 + const spoolDir = await mkdtemp(join(tmpdir(), "pi-firefox-bench-"));
248 + return {
249 + ws,
250 + ctx: {
251 + pending,
252 + ledger: { subscriptions: [], intercepts: [], preloadScripts: [] },
253 + spoolDir,
254 + },
255 + frames,
256 + byId: new Map(),
257 + };
260 +export async function makeSpoolCtx(): Promise<BatchCtx> {
261 + const dir = await mkdtemp(join(tmpdir(), "pi-firefox-bench-"));
262 + return {
263 + pending: new Map(),
264 + ledger: { subscriptions: [], intercepts: [], preloadScripts: [] },
265 + spoolDir: dir,
266 + };

extensions/firefox-bidi/__bench__/e2e.mts

Added
new file mode 100644
1 +// End-to-end benchmark against a real headless Firefox: measures the batch
2 +// loop where spool writes can overlap real WebSocket round-trips.
3 +// Baseline = inline spool awaits (__bench__/baseline.ts), candidate = deferred
4 +// spool drain (lib.ts). One sessionless /session connection carries both
5 +// variants; browsing contexts are created up front. Interleaved pairs,
6 +// Wilcoxon signed-rank.
7 +// Usage: node --experimental-strip-types __bench__/e2e.mts [pairs]
9 +import { spawn } from "node:child_process";
10 +import { mkdtemp, rm } from "node:fs/promises";
11 +import { tmpdir } from "node:os";
12 +import { join } from "node:path";
13 +import { performance } from "node:perf_hooks";
14 +import { runBatchBaseline } from "./baseline.ts";
15 +import { runBatch, type BidiResponse, type BatchCtx, type Frame } from "../lib.ts";
17 +const ITER = Number(process.argv[2] ?? 12);
19 +function stats(s: number[]) {
20 + const sorted = [...s].sort((a, b) => a - b);
21 + const mean = s.reduce((a, b) => a + b, 0) / s.length;
22 + return { n: s.length, median: sorted[Math.floor(sorted.length / 2)], mean, sd: Math.sqrt(s.reduce((a, b) => a + (b - mean) ** 2, 0) / s.length) };
24 +function erf(x: number): number {
25 + const t = 1 / (1 + 0.3275911 * Math.abs(x));
26 + const y = 1 - ((((1.061405429 * t - 1.453152027) * t + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t * Math.exp(-x * x);
27 + return x >= 0 ? y : -y;
29 +function wilcoxon(d: number[]): number {
30 + const nz = d.filter((x) => x !== 0).map((x) => ({ abs: Math.abs(x), sign: Math.sign(x), rank: 0 }));
31 + nz.sort((a, b) => a.abs - b.abs);
32 + nz.forEach((o, i) => (o.rank = i + 1));
33 + const w = nz.filter((o) => o.sign > 0).reduce((a, o) => a + o.rank, 0);
34 + const n = nz.length;
35 + const z = (w - (n * (n + 1)) / 4) / Math.sqrt((n * (n + 1) * (2 * n + 1)) / 24);
36 + return 2 * (1 - 0.5 * (1 + erf(Math.abs(z) / Math.SQRT2)));
39 +// Launch Firefox
40 +const profileDir = await mkdtemp(join(tmpdir(), "ffx-e2e-"));
41 +const child = spawn(process.env.FIREFOX_BIN ?? "firefox", ["--headless", "--no-remote", "--profile", profileDir, "--remote-debugging-port", "0"], { stdio: ["ignore", "pipe", "pipe"] });
42 +const baseUrl = await new Promise<string>((resolve, reject) => {
43 + const t = setTimeout(() => reject(new Error("no listener")), 30_000);
44 + const grab = (buf: Buffer) => {
45 + const m = String(buf).match(/WebDriver BiDi listening on (ws:\/\/\S+)/);
46 + if (m) {
47 + clearTimeout(t);
48 + resolve(m[1]);
49 + }
50 + };
51 + child.stdout?.on("data", grab);
52 + child.stderr?.on("data", grab);
53 +});
55 +// One sessionless connection; session.new attaches it. All commands and both
56 +// variants run through this socket.
57 +const ws = await connect(`${baseUrl}/session`);
58 +const pending = new Map<number, (r: BidiResponse) => void>();
59 +ws.onmessage = (m) => {
60 + const f = JSON.parse(String(m.data));
61 + if (f.type === "event") return;
62 + pending.get(f.id)?.(f);
63 +};
64 +function connect(url: string): Promise<WebSocket> {
65 + return new Promise((resolve, reject) => {
66 + const w = new WebSocket(url);
67 + const t = setTimeout(() => reject(new Error("connect timeout")), 10_000);
68 + w.onopen = () => {
69 + clearTimeout(t);
70 + resolve(w);
71 + };
72 + w.onerror = () => {
73 + clearTimeout(t);
74 + reject(new Error("connect failed"));
75 + };
76 + });
78 +let nextId = 1;
79 +function call(method: string, params: Record<string, unknown>): Promise<BidiResponse> {
80 + const id = nextId++;
81 + return new Promise((resolve) => {
82 + pending.set(id, resolve);
83 + ws.send(JSON.stringify({ id, method, params }));
84 + });
87 +const sn = await call("session.new", { capabilities: { alwaysMatch: { acceptInsecureCerts: true } } });
88 +if (sn.type !== "success") throw new Error(`session.new failed: ${JSON.stringify(sn)}`);
90 +async function makeSlot(spoolParent: string): Promise<{ ctx: BatchCtx; ctxId: string; nextId: () => number; n: { v: number } }> {
91 + const cr = await call("browsingContext.create", { type: "tab" });
92 + const ctxId = (cr.result as { context: string }).context;
93 + const spoolDir = await mkdtemp(join(tmpdir(), spoolParent));
94 + const n = { v: 500 };
95 + return {
96 + ctx: { pending, ledger: { subscriptions: [], intercepts: [], preloadScripts: [] }, spoolDir },
97 + ctxId,
98 + nextId: () => n.v++,
99 + n,
100 + };
103 +const framesFor = (ctxId: string, ids: () => number): Frame[] => [
104 + { id: ids(), method: "browsingContext.navigate", params: { context: ctxId, url: "data:text/html,<b>x</b>", wait: "complete" } },
105 + { id: ids(), method: "script.evaluate", params: { expression: `"A".repeat(${300 * 1024})`, awaitPromise: false, target: { context: ctxId } } },
106 + { id: ids(), method: "script.evaluate", params: { expression: `"A".repeat(${300 * 1024})`, awaitPromise: false, target: { context: ctxId } } },
107 + { id: ids(), method: "script.evaluate", params: { expression: `"A".repeat(${300 * 1024})`, awaitPromise: false, target: { context: ctxId } } },
108 + { id: ids(), method: "browsingContext.navigate", params: { context: ctxId, url: "data:text/html,<i>y</i>", wait: "complete" } },
111 +const slotA = await makeSlot("ffx-e2e-a-");
112 +const slotB = await makeSlot("ffx-e2e-b-");
114 +// Warmup
115 +await runBatchBaseline(ws, slotA.ctx, framesFor(slotA.ctxId, slotA.nextId), new Map(), 30000);
116 +await runBatch(ws, slotB.ctx, framesFor(slotB.ctxId, slotB.nextId), new Map(), 30000);
118 +const pairs: { base: number; cand: number }[] = [];
119 +for (let i = 0; i < ITER; i++) {
121 + const t0 = performance.now();
122 + await runBatchBaseline(ws, slotA.ctx, framesFor(slotA.ctxId, slotA.nextId), new Map(), 30000);
123 + pairs.push({ base: performance.now() - t0, cand: 0 });
126 + const t1 = performance.now();
127 + await runBatch(ws, slotB.ctx, framesFor(slotB.ctxId, slotB.nextId), new Map(), 30000);
128 + pairs[i].cand = performance.now() - t1;
132 +const baseSamples = pairs.map((p) => p.base);
133 +const candSamples = pairs.map((p) => p.cand);
134 +const diffs = pairs.map((p) => p.base - p.cand);
135 +console.log(JSON.stringify({
136 + pairs: ITER,
137 + samples_ms: { baseline_inlineSpool: baseSamples, candidate_deferredSpool: candSamples },
138 + stats: { baseline: stats(baseSamples), candidate: stats(candSamples) },
139 + paired_diff_ms: { mean: diffs.reduce((a, b) => a + b, 0) / diffs.length },
140 + wilcoxon_p: wilcoxon(diffs),
141 +}));
143 +ws.close();
144 +child.kill("SIGTERM");
145 +await rm(profileDir, { recursive: true, force: true }).catch(() => {});

extensions/firefox-bidi/__bench__/runBatch.bench.mts

Added
new file mode 100644
1 +// Microbenchmark: batch-of-25 frames (22 small results + 3 x 512KB results)
2 +// through the batch-processing hot path, plus final payload serialization.
3 +// Usage: npx tsx bench/runBatch.bench.mts [iterations]
4 +// Prints JSON: { iterations, samples_ms, stats }
6 +import { performance } from "node:perf_hooks";
7 +import { runBatchBaseline, payloadBaseline, makeFixture } from "./baseline.ts";
8 +import { runBatch, assemblePayload, type BidiResponse, type Frame, type BatchCtx } from "../lib.ts";
10 +const ITER = Number(process.argv[2] ?? 15);
12 +function stats(samples: number[]) {
13 + const sorted = [...samples].sort((a, b) => a - b);
14 + const mean = samples.reduce((a, b) => a + b, 0) / samples.length;
15 + const sd = Math.sqrt(samples.reduce((a, b) => a + (b - mean) ** 2, 0) / samples.length);
16 + return { n: samples.length, median: sorted[Math.floor(sorted.length / 2)], mean, sd, min: sorted[0], max: sorted[sorted.length - 1], p90: sorted[Math.floor(sorted.length * 0.9)] };
19 +// Warmup: JIT + fs settle
21 + const fx = await makeFixture();
22 + await runBatchBaseline(fx.ws, fx.ctx, fx.frames, fx.byId, 30000);
23 + const fx2 = await makeFixture();
24 + await runBatch(fx2.ws, fx2.ctx, fx2.frames as Frame[], fx2.byId, 30000);
27 +// Interleaved A/B pairs cancel machine drift; per-pair differences feed the
28 +// signed-rank test. Fixture construction (fs mkdir) stays outside the timer.
29 +const pairs: { base: number; cand: number }[] = [];
30 +for (let i = 0; i < ITER; i++) {
31 + const fx = await makeFixture();
32 + const t0 = performance.now();
33 + const base = await runBatchBaseline(fx.ws, fx.ctx, fx.frames, fx.byId, 30000);
34 + const baseText = payloadBaseline({ sessionId: "s", port: 9222, capabilities: {} }, base, [], undefined, 200);
35 + pairs.push({ base: performance.now() - t0, cand: 0 });
37 + const fx2 = await makeFixture();
38 + const t1 = performance.now();
39 + const cand = await runBatch(fx2.ws, fx2.ctx, fx2.frames as Frame[], fx2.byId, 30000);
40 + const candText = assemblePayload({ sessionId: "s", port: 9222, capabilities: {} }, cand.texts, [], 200, undefined);
41 + pairs[i].cand = performance.now() - t1;
42 + if (i === 0) {
43 + const norm = (t: string, dir: string) => JSON.stringify(JSON.parse(t.replaceAll(dir, "<SPOOL>")));
44 + if (norm(baseText, fx.ctx.spoolDir) !== norm(candText, fx2.ctx.spoolDir)) {
45 + console.error("OUTPUT MISMATCH between baseline and candidate");
46 + process.exit(1);
47 + }
48 + }
51 +const baseSamples = pairs.map((p) => p.base);
52 +const candSamples = pairs.map((p) => p.cand);
53 +const diffs = pairs.map((p) => p.base - p.cand);
55 +// Wilcoxon signed-rank (normal approximation, no zero-diff correction)
56 +function wilcoxon(d: number[]): number {
57 + const nz = d.filter((x) => x !== 0).map((x, i) => ({ abs: Math.abs(x), sign: Math.sign(x), rank: 0 }));
58 + nz.sort((a, b) => a.abs - b.abs);
59 + nz.forEach((o, i) => (o.rank = i + 1));
60 + const wPlus = nz.filter((o) => o.sign > 0).reduce((a, o) => a + o.rank, 0);
61 + const n = nz.length;
62 + const mu = (n * (n + 1)) / 4;
63 + const sigma = Math.sqrt((n * (n + 1) * (2 * n + 1)) / 24);
64 + const z = (wPlus - mu) / sigma;
65 + const phi = 0.5 * (1 + erf(Math.abs(z) / Math.SQRT2));
66 + return 2 * (1 - phi);
68 +function erf(x: number): number {
69 + const t = 1 / (1 + 0.3275911 * Math.abs(x));
70 + const y = 1 - ((((1.061405429 * t - 1.453152027) * t + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t * Math.exp(-x * x);
71 + return x >= 0 ? y : -y;
74 +console.log(JSON.stringify({
75 + iterations: ITER,
76 + samples_ms: { baseline: baseSamples, candidate: candSamples },
77 + paired_diff_ms: { mean: diffs.reduce((a, b) => a + b, 0) / diffs.length },
78 + stats: { baseline: stats(baseSamples), candidate: stats(candSamples) },
79 + wilcoxon_p: wilcoxon(diffs),
80 + output_equal: true,
81 +}));

extensions/firefox-bidi/__tests__/harness.test.ts

Added
new file mode 100644
1 +import { mkdtempSync } from "node:fs";
2 +import { chmod, writeFile } from "node:fs/promises";
3 +import { tmpdir } from "node:os";
4 +import path from "node:path";
5 +import { afterEach, describe, expect, it } from "vitest";
6 +import { createTestSession, type TestSession } from "../../../test/harness";
8 +const extensionPath = path.resolve(import.meta.dirname, "../index.ts");
10 +describe("firefox-bidi pi-test-harness", () => {
11 + let t: TestSession | undefined;
13 + afterEach(() => {
14 + t?.dispose();
15 + t = undefined;
16 + });
18 + it("loads the tool inert, the command with completions, and the bundled skill", async () => {
19 + t = await createTestSession({ extensions: [extensionPath] });
20 + const runner = t.session.extensionRunner;
21 + expect(runner.extensions).toHaveLength(1);
22 + expect(runner.extensions[0]?.path).toBe(extensionPath);
24 + const tool = runner.extensions[0]?.tools.get("firefox_bidi")?.definition;
25 + expect(tool).toBeDefined();
26 + expect(tool?.description).toContain("WebDriver BiDi");
27 + expect(tool?.promptGuidelines).toHaveLength(2);
29 + // tool ships inert: not advertised until /firefox
30 + expect(t.session.getActiveToolNames()).not.toContain("firefox_bidi");
31 + expect(t.session.systemPrompt).not.toContain(
32 + "Drive Firefox with raw WebDriver BiDi frames",
33 + );
35 + const command = runner.getCommand("firefox");
36 + expect(command).toBeDefined();
37 + expect(command?.getArgumentCompletions?.("")).toEqual([
38 + { value: "close", label: "close" },
39 + ]);
40 + expect(command?.getArgumentCompletions?.("clo")).toEqual([
41 + { value: "close", label: "close" },
42 + ]);
43 + expect(command?.getArgumentCompletions?.("x")).toBeNull();
45 + const skill = t.session.resourceLoader
46 + .getSkills()
47 + .skills.find((candidate) => candidate.name === "firefox-bidi");
48 + expect(skill).toBeDefined();
49 + });
51 + it("activates via /firefox and deactivates via /firefox close without launching", async () => {
52 + t = await createTestSession({
53 + extensions: [extensionPath],
54 + env: { FIREFOX_BIN: "/nonexistent/firefox-bin" },
55 + });
56 + const runner = t.session.extensionRunner;
57 + const command = runner.getCommand("firefox");
58 + const context = {
59 + ...runner.createCommandContext(),
60 + mode: "print" as const,
61 + hasUI: false,
62 + };
64 + expect(t.session.getActiveToolNames()).not.toContain("firefox_bidi");
65 + await command?.handler("", context);
66 + expect(t.session.getActiveToolNames()).toContain("firefox_bidi");
67 + await command?.handler("close", context);
68 + expect(t.session.getActiveToolNames()).not.toContain("firefox_bidi");
69 + });
71 + it("activates on session_start with the PI_FIREFOX=1 dev hook", async () => {
72 + t = await createTestSession({
73 + extensions: [extensionPath],
74 + env: { PI_FIREFOX: "1", FIREFOX_BIN: "/nonexistent/firefox-bin" },
75 + });
76 + expect(t.session.getActiveToolNames()).toContain("firefox_bidi");
77 + });
79 + it("fails fast with a clear error when the firefox binary is missing", async () => {
80 + t = await createTestSession({
81 + extensions: [extensionPath],
82 + env: { FIREFOX_BIN: "/nonexistent/firefox-bin" },
83 + });
84 + const runner = t.session.extensionRunner;
85 + const command = runner.getCommand("firefox");
86 + const context = {
87 + ...runner.createCommandContext(),
88 + mode: "print" as const,
89 + hasUI: false,
90 + };
91 + await command?.handler("", context);
93 + const tool = runner.extensions[0]?.tools.get("firefox_bidi")?.definition;
94 + await expect(
95 + tool?.execute(
96 + "call-1",
97 + { frames: [{ method: "session.status" }] },
98 + undefined,
99 + undefined,
100 + context,
101 + ),
102 + ).rejects.toThrow(/firefox not found/);
103 + });
105 + it.runIf(process.platform !== "win32")(
106 + "surfaces immediate browser exit through a controlled fake firefox on PATH",
107 + async () => {
108 + const fakeDir = mkdtempSync(path.join(tmpdir(), "pi-firefox-fake-"));
109 + const fakeBin = path.join(fakeDir, "fake-firefox");
110 + await writeFile(fakeBin, "#!/bin/sh\nexit 7\n");
111 + await chmod(fakeBin, 0o755);
113 + t = await createTestSession({
114 + extensions: [extensionPath],
115 + env: { FIREFOX_BIN: fakeBin },
116 + });
117 + const runner = t.session.extensionRunner;
118 + const command = runner.getCommand("firefox");
119 + const context = {
120 + ...runner.createCommandContext(),
121 + mode: "print" as const,
122 + hasUI: false,
123 + };
124 + await command?.handler("", context);
126 + const tool = runner.extensions[0]?.tools.get("firefox_bidi")?.definition;
127 + await expect(
128 + tool?.execute(
129 + "call-2",
130 + { frames: [{ method: "session.status" }] },
131 + undefined,
132 + undefined,
133 + context,
134 + ),
135 + ).rejects.toThrow(/firefox exited immediately \(code 7\)/);
136 + },
137 + );
138 +});

extensions/firefox-bidi/__tests__/lib.test.ts

Added
new file mode 100644
1 +import assert from "node:assert/strict";
2 +import { mkdtemp, readdir } from "node:fs/promises";
3 +import { tmpdir } from "node:os";
4 +import { join } from "node:path";
5 +import { test } from "vitest";
6 +import {
7 + assemblePayload,
8 + interpolate,
9 + ledgerUpdate,
10 + parseScreencastSettings,
11 + resolveScreencastDestinations,
12 + runBatch,
13 +} from "../lib.ts";
15 +const res = (id: number, result: unknown) => ({ id, type: "success", result });
17 +test("interpolate resolves primitive refs from earlier responses", () => {
18 + const byId = new Map([[2, res(2, { context: "abc-1" })]]);
19 + assert.equal(interpolate("{{2.result.context}}", byId), "abc-1");
20 +});
22 +test("interpolate resolves deep paths inside nested params", () => {
23 + const byId = new Map([[4, res(4, { screencast: "sc-9" })]]);
24 + const params = {
25 + context: "{{2.result.context}}",
26 + nested: { a: ["{{4.result.screencast}}", "plain"] },
27 + };
28 + const out = interpolate(
29 + params,
30 + new Map([
31 + [2, res(2, { context: "ctx-7" })],
32 + [4, byId.get(4)],
33 + ]),
34 + );
35 + assert.deepEqual(out, { context: "ctx-7", nested: { a: ["sc-9", "plain"] } });
36 +});
38 +test("interpolate leaves strings without refs untouched", () => {
39 + assert.equal(
40 + interpolate("data:text/html,<h1>hi</h1>", new Map()),
41 + "data:text/html,<h1>hi</h1>",
42 + );
43 +});
45 +test("interpolate throws on unknown frame id", () => {
46 + assert.throws(
47 + () => interpolate("{{9.result.x}}", new Map()),
48 + /no successful response/,
49 + );
50 +});
52 +test("interpolate throws on non-primitive target", () => {
53 + const byId = new Map([[1, res(1, { tree: { a: 1 } })]]);
54 + assert.throws(() => interpolate("{{1.result.tree}}", byId), /primitive/);
55 +});
57 +test("parseScreencastSettings accepts string dirs, rejects the rest", () => {
58 + assert.equal(parseScreencastSettings({ screencastDir: "/x" }), "/x");
59 + assert.equal(parseScreencastSettings({ screencastDir: " " }), undefined);
60 + assert.equal(parseScreencastSettings({ screencastDir: 42 }), undefined);
61 + assert.equal(parseScreencastSettings("nope"), undefined);
62 + assert.equal(parseScreencastSettings(null), undefined);
63 +});
65 +test("resolveScreencastDestinations creates missing dir and injects explicit param", async () => {
66 + const dir = join(await mkdtemp(join(tmpdir(), "ffx-test-")), "videos");
67 + const frames = [
68 + {
69 + method: "browsingContext.startScreencast",
70 + params: { context: "c", destinationFolder: dir },
71 + },
72 + ];
73 + await resolveScreencastDestinations(frames);
74 + assert.equal(frames[0].params?.destinationFolder, dir);
75 + assert.equal((await readdir(dir)).length, 0);
76 +});
78 +test("resolveScreencastDestinations falls back to configured callback when param absent", async () => {
79 + const dir = join(await mkdtemp(join(tmpdir(), "ffx-test-")), "from-settings");
80 + const frames = [
81 + { method: "browsingContext.startScreencast", params: { context: "c" } },
82 + ];
83 + await resolveScreencastDestinations(frames, () => dir);
84 + assert.equal(frames[0].params?.destinationFolder, dir);
85 + assert.ok(await readdir(dir));
86 +});
88 +test("resolveScreencastDestinations explicit param beats configured", async () => {
89 + const dir = join(await mkdtemp(join(tmpdir(), "ffx-test-")), "explicit");
90 + const frames = [
91 + {
92 + method: "browsingContext.startScreencast",
93 + params: { context: "c", destinationFolder: dir },
94 + },
95 + ];
96 + await resolveScreencastDestinations(frames, () => "/from-settings");
97 + assert.equal(frames[0].params?.destinationFolder, dir);
98 +});
100 +test("resolveScreencastDestinations leaves params untouched when nothing configured", async () => {
101 + const frames = [
102 + { method: "browsingContext.startScreencast", params: { context: "c" } },
103 + ];
104 + await resolveScreencastDestinations(frames);
105 + assert.equal(frames[0].params?.destinationFolder, undefined);
106 +});
108 +test("resolveScreencastDestinations ignores other methods", async () => {
109 + const frames = [{ method: "browsingContext.navigate", params: {} }];
110 + await resolveScreencastDestinations(frames);
111 + assert.deepEqual(frames[0].params, {});
112 +});
114 +interface Ctx {
115 + pending: Map<
116 + number,
117 + (r: { id: number; type?: string; result?: unknown }) => void
118 + >;
119 + ledger: {
120 + subscriptions: string[];
121 + intercepts: string[];
122 + preloadScripts: string[];
123 + };
124 + spoolDir: string;
127 +async function ctx(): Promise<Ctx> {
128 + return {
129 + pending: new Map(),
130 + ledger: { subscriptions: [], intercepts: [], preloadScripts: [] },
131 + spoolDir: await mkdtemp(join(tmpdir(), "ffx-rb-")),
132 + };
135 +test("runBatch resolves frames in order and tracks ledger + byId", async () => {
136 + const c = await ctx();
137 + const replies = new Map<number, unknown>([
138 + [1, {}],
139 + [2, { intercept: "int-1" }],
140 + ]);
141 + const pending = new Map<number, (r: any) => void>();
142 + const ws = {
143 + send: (s: string) => {
144 + const f = JSON.parse(s);
145 + queueMicrotask(() =>
146 + pending.get(f.id)?.({
147 + id: f.id,
148 + type: "success",
149 + result: replies.get(f.id),
150 + }),
151 + );
152 + },
153 + };
154 + const b = { pending, ledger: c.ledger, spoolDir: c.spoolDir };
155 + const byId = new Map();
156 + const frames = [
157 + { id: 1, method: "session.subscribe", params: { events: ["network"] } },
158 + { id: 2, method: "network.addIntercept", params: { phases: ["request"] } },
159 + ];
160 + const { responses } = await runBatch(
161 + ws as unknown as WebSocket,
162 + b as never,
163 + frames as never,
164 + byId,
165 + 5000,
166 + );
167 + assert.equal(responses.length, 2);
168 + assert.deepEqual(
169 + responses.map((r) => r.id),
170 + [1, 2],
171 + );
172 + assert.ok(responses.every((r) => r.type === "success"));
173 + assert.deepEqual(c.ledger.subscriptions, ["network"]);
174 + assert.deepEqual(c.ledger.intercepts, ["int-1"]);
175 + assert.ok(byId.get(2));
176 +});
178 +test("runBatch aborts remaining frames after interpolation error", async () => {
179 + const pending = new Map<number, (r: any) => void>();
180 + const ws = {
181 + send: (s: string) => {
182 + const f = JSON.parse(s);
183 + queueMicrotask(() =>
184 + pending.get(f.id)?.({ id: f.id, type: "success", result: {} }),
185 + );
186 + },
187 + };
188 + const b = {
189 + pending,
190 + ledger: { subscriptions: [], intercepts: [], preloadScripts: [] },
191 + spoolDir: await mkdtemp(join(tmpdir(), "ffx-rb-")),
192 + };
193 + const byId = new Map();
194 + const frames = [
195 + { id: 1, method: "browsingContext.create", params: {} },
197 + id: 2,
198 + method: "browsingContext.navigate",
199 + params: { context: "{{9.result.context}}" },
200 + },
201 + { id: 3, method: "script.evaluate", params: {} },
202 + ];
203 + const { responses } = await runBatch(
204 + ws as unknown as WebSocket,
205 + b as never,
206 + frames as never,
207 + byId,
208 + 5000,
209 + );
210 + assert.equal(responses[1].type, "error");
211 + assert.equal(responses[2].skipped, true);
212 +});
214 +test("runBatch spools oversized results and replaces them inline", async () => {
215 + const dir = await mkdtemp(join(tmpdir(), "ffx-rb-"));
216 + const pending = new Map<number, (r: any) => void>();
217 + const ws = {
218 + send: (s: string) => {
219 + const f = JSON.parse(s);
220 + queueMicrotask(() =>
221 + pending.get(f.id)?.({
222 + id: f.id,
223 + type: "success",
224 + result: { data: "A".repeat(300 * 1024) },
225 + }),
226 + );
227 + },
228 + };
229 + const b = {
230 + pending,
231 + ledger: { subscriptions: [], intercepts: [], preloadScripts: [] },
232 + spoolDir: dir,
233 + };
234 + const { responses } = await runBatch(
235 + ws as unknown as WebSocket,
236 + b as never,
238 + { id: 1, method: "browsingContext.captureScreenshot", params: {} },
239 + ] as never,
240 + new Map(),
241 + 5000,
242 + );
243 + const spooled = responses[0].result as { spooled: string };
244 + assert.ok(spooled.spooled.startsWith(dir));
245 + assert.equal((await readdir(dir)).length, 1);
246 +});
248 +test("runBatch flushes deferred spool writes before returning", async () => {
249 + const dir = await mkdtemp(join(tmpdir(), "ffx-rb-"));
250 + const pending = new Map<number, (r: any) => void>();
251 + const ws = {
252 + send: (s: string) => {
253 + const f = JSON.parse(s);
254 + queueMicrotask(() =>
255 + pending.get(f.id)?.({
256 + id: f.id,
257 + type: "success",
258 + result: { data: "A".repeat(300 * 1024) },
259 + }),
260 + );
261 + },
262 + };
263 + const b = {
264 + pending,
265 + ledger: { subscriptions: [], intercepts: [], preloadScripts: [] },
266 + spoolDir: dir,
267 + };
268 + const { responses } = await runBatch(
269 + ws as unknown as WebSocket,
270 + b as never,
272 + { id: 1, method: "browsingContext.captureScreenshot", params: {} },
273 + { id: 2, method: "browsingContext.captureScreenshot", params: {} },
274 + ] as never,
275 + new Map(),
276 + 5000,
277 + );
278 + assert.equal(responses.length, 2);
279 + assert.equal(
280 + (await readdir(dir)).length,
281 + 2,
282 + "both spool files flushed before return",
283 + );
284 +});
286 +test("assemblePayload produces JSON identical to object stringify", () => {
287 + const session = {
288 + sessionId: "s",
289 + port: 9222,
290 + capabilities: { browserName: "firefox" },
291 + };
292 + const responses = [
294 + id: 1,
295 + method: "session.new",
296 + type: "success",
297 + result: { sessionId: "s" },
298 + },
300 + id: 2,
301 + method: "browsingContext.create",
302 + type: "error",
303 + error: "x",
304 + message: "y",
305 + },
306 + ];
307 + const events = [
308 + { type: "event", method: "log.entryAdded", params: { a: 1 } },
309 + ];
310 + const wait = { timedOut: true, method: "x" };
311 + const got = assemblePayload(
312 + session,
313 + responses.map((r) => JSON.stringify(r)),
314 + events,
315 + 200,
316 + wait,
317 + );
318 + const want = JSON.stringify({ session, responses, events, wait });
319 + assert.equal(JSON.parse(got).responses[0].result.sessionId, "s");
320 + assert.deepEqual(JSON.parse(got), JSON.parse(want));
321 +});
323 +test("ledgerUpdate removes intercepts and preload scripts", () => {
324 + const ledger = {
325 + subscriptions: [],
326 + intercepts: ["i1", "i2"],
327 + preloadScripts: ["p1"],
328 + };
329 + ledgerUpdate(
330 + { pending: new Map(), ledger, spoolDir: "" } as never,
331 + "network.removeIntercept",
332 + { intercepts: ["i1"] },
333 + { id: 1, type: "success" },
334 + );
335 + ledgerUpdate(
336 + { pending: new Map(), ledger, spoolDir: "" } as never,
337 + "script.removePreloadScript",
338 + { id: "p1" },
339 + { id: 2, type: "success" },
340 + );
341 + assert.deepEqual(ledger.intercepts, ["i2"]);
342 + assert.deepEqual(ledger.preloadScripts, []);
343 +});

extensions/firefox-bidi/index.ts

Added
new file mode 100644
1 +/**
2 + * Firefox BiDi extension: raw WebDriver BiDi frames from the agent.
3 + *
4 + * - `firefox_bidi` tool: batch of raw {id?, method, params} frames, verbatim
5 + * BiDi responses + events. Inactive until `/firefox`.
6 + * - `/firefox` command: activates the tool (launches Firefox lazily).
7 + * - `/firefox close`: shuts browser down and deactivates the tool.
8 + *
9 + * Firefox is resolved from `FIREFOX_BIN` or PATH; a missing binary is
10 + * reported with a warning and every launch uses a fresh temp profile.
11 + */
13 +import { type ChildProcess, spawn } from "node:child_process";
14 +import { accessSync, constants as fsConstants, statSync } from "node:fs";
15 +import { appendFile, mkdtemp, rm, writeFile } from "node:fs/promises";
16 +import { tmpdir } from "node:os";
17 +import { delimiter, join } from "node:path";
18 +import { fileURLToPath } from "node:url";
19 +import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
20 +import { SettingsManager } from "@earendil-works/pi-coding-agent";
21 +import type { AutocompleteItem } from "@earendil-works/pi-tui";
22 +import { Type } from "typebox";
23 +import {
24 + assemblePayload,
25 + type BidiResponse,
26 + type Frame,
27 + parseScreencastSettings,
28 + resolveScreencastDestinations,
29 + runBatch,
30 +} from "./lib.ts";
32 +// ── state (one browser per Pi session) ──────────────────────────────────────
34 +interface BidiEvent {
35 + type: "event";
36 + method: string;
37 + params: unknown;
40 +interface Ledger {
41 + subscriptions: string[];
42 + intercepts: string[];
43 + preloadScripts: string[];
46 +interface Browser {
47 + child: ChildProcess;
48 + port: number;
49 + profileDir: string;
50 + spoolDir: string;
51 + access: boolean;
52 + sessionId: string | null;
53 + capabilities: Record<string, unknown> | null;
54 + ws: WebSocket | null;
55 + nextId: number;
56 + pending: Map<number, (r: BidiResponse) => void>;
57 + waiters: Array<{
58 + test: (e: BidiEvent) => boolean;
59 + resolve: (e: BidiEvent) => void;
60 + }>;
61 + events: BidiEvent[];
62 + ledger: Ledger;
65 +let browser: Browser | null = null;
66 +// Previous batch's responses by frame id, so {{N.result.path}} works across calls.
67 +let lastById = new Map<number, BidiResponse>();
68 +const EVENT_CAP = 200;
69 +const CONNECT_TIMEOUT = 10_000;
70 +const DEFAULT_FRAME_TIMEOUT = 30_000;
72 +const SKILL_PATH = fileURLToPath(
73 + new URL("./skills/firefox-bidi/SKILL.md", import.meta.url),
74 +);
76 +// ── binary resolution + opt-in file diagnostics ─────────────────────────────
78 +function resolveOnPath(name: string): string | undefined {
79 + const extensions = process.platform === "win32" ? [".exe", ".cmd", ""] : [""];
80 + for (const dir of (process.env.PATH ?? "").split(delimiter)) {
81 + if (!dir) continue;
82 + for (const ext of extensions) {
83 + const candidate = join(dir, name + ext);
84 + try {
85 + accessSync(
86 + candidate,
87 + process.platform === "win32" ? fsConstants.F_OK : fsConstants.X_OK,
88 + );
89 + if (statSync(candidate).isFile()) return candidate;
90 + } catch {
91 + // Keep searching PATH.
92 + }
93 + }
94 + }
95 + return undefined;
98 +function firefoxBinary(): string {
99 + return process.env.FIREFOX_BIN ?? resolveOnPath("firefox") ?? "firefox";
102 +function assertBinaryAvailable(bin: string): void {
103 + try {
104 + accessSync(
105 + bin,
106 + process.platform === "win32" ? fsConstants.F_OK : fsConstants.X_OK,
107 + );
108 + } catch {
109 + throw new Error(
110 + `firefox not found: ${bin} — set FIREFOX_BIN or install Firefox on PATH`,
111 + );
115 +function debugLog(line: string): void {
116 + if (!process.env.PI_FIREFOX_DEBUG) return;
117 + void appendFile(
118 + join(tmpdir(), "pi-firefox-bidi-debug.log"),
119 + `${new Date().toISOString()} ${line}\n`,
120 + ).catch(() => {});
123 +// ── websocket plumbing ──────────────────────────────────────────────────────
125 +function connect(url: string, retries = 0): Promise<WebSocket> {
126 + return new Promise((resolve, reject) => {
127 + const ws = new WebSocket(url);
128 + const t = setTimeout(() => {
129 + ws.close();
130 + reject(new Error(`connect timeout: ${url}`));
131 + }, CONNECT_TIMEOUT);
132 + ws.onopen = () => {
133 + clearTimeout(t);
134 + resolve(ws);
135 + };
136 + ws.onerror = () => {
137 + clearTimeout(t);
138 + if (retries > 0)
139 + setTimeout(() => connect(url, retries - 1).then(resolve, reject), 300);
140 + else reject(new Error(`connect failed: ${url}`));
141 + };
142 + });
145 +function attachSocket(b: Browser, ws: WebSocket): void {
146 + b.ws = ws;
147 + ws.onmessage = (m) => {
148 + let frame: Record<string, unknown>;
149 + try {
150 + frame = JSON.parse(String(m.data));
151 + } catch {
152 + return;
154 + if (frame.type === "event") {
155 + const ev = frame as unknown as BidiEvent;
156 + b.events.push(ev);
157 + if (b.events.length > EVENT_CAP * 4)
158 + b.events.splice(0, b.events.length - EVENT_CAP);
159 + b.waiters = b.waiters.filter((w) => {
160 + if (w.test(ev)) {
161 + w.resolve(ev);
162 + return false;
164 + return true;
165 + });
166 + return;
168 + const id = frame.id as number;
169 + const r = b.pending.get(id);
170 + if (r) {
171 + b.pending.delete(id);
172 + r(frame as unknown as BidiResponse);
174 + };
175 + ws.onclose = () => {
176 + if (b.ws === ws) b.ws = null;
177 + for (const [, r] of b.pending) r({ id: -1, type: "connection-closed" });
178 + b.pending.clear();
179 + };
182 +async function sessionSocket(b: Browser): Promise<WebSocket> {
183 + if (b.ws && b.ws.readyState === WebSocket.OPEN) return b.ws;
184 + if (b.sessionId) {
185 + try {
186 + const ws = await connect(
187 + `ws://127.0.0.1:${b.port}/session/${b.sessionId}`,
188 + 3,
189 + );
190 + attachSocket(b, ws);
191 + return ws;
192 + } catch {
193 + b.sessionId = null; // stale id, browser still alive
196 + const ws = await connect(`ws://127.0.0.1:${b.port}/session`, 3);
197 + attachSocket(b, ws);
198 + return ws;
201 +// ── launch / teardown ───────────────────────────────────────────────────────
203 +async function launch(opts: {
204 + headed?: boolean;
205 + access?: boolean;
206 + prefs?: string[];
207 +}): Promise<Browser> {
208 + const bin = firefoxBinary();
209 + assertBinaryAvailable(bin);
210 + const profileDir = await mkdtemp(join(tmpdir(), "pi-firefox-"));
211 + const spoolDir = await mkdtemp(join(tmpdir(), "pi-firefox-spool-"));
212 + if (opts.prefs?.length) {
213 + const userJs = opts.prefs
214 + .map((p) => {
215 + const eq = p.indexOf("=");
216 + if (eq <= 0) throw new Error(`--pref must be key=value, got ${p}`);
217 + const k = p.slice(0, eq);
218 + let v: unknown = p.slice(eq + 1);
219 + try {
220 + v = JSON.parse(String(v));
221 + } catch {
222 + /* keep string */
224 + return `user_pref(${JSON.stringify(k)}, ${JSON.stringify(v)});`;
225 + })
226 + .join("\n");
227 + await writeFile(join(profileDir, "user.js"), userJs + "\n");
229 + const args = [
230 + "--headless",
231 + "--no-remote",
232 + "--profile",
233 + profileDir,
234 + "--remote-debugging-port",
235 + "0",
236 + ];
237 + if (opts.headed) args.shift();
238 + if (opts.access) args.push("--remote-allow-system-access");
239 + const child = spawn(bin, args, { stdio: ["ignore", "pipe", "pipe"] });
240 + const url = await new Promise<string>((resolve, reject) => {
241 + const t = setTimeout(
242 + () => reject(new Error("firefox: no BiDi listener within 30s")),
243 + 30_000,
244 + );
245 + const grab = (buf: Buffer) => {
246 + const m = String(buf).match(/WebDriver BiDi listening on (ws:\/\/\S+)/);
247 + if (m) {
248 + clearTimeout(t);
249 + resolve(m[1]);
251 + };
252 + child.stdout?.on("data", grab);
253 + child.stderr?.on("data", grab);
254 + child.on("exit", (code) => {
255 + clearTimeout(t);
256 + reject(new Error(`firefox exited immediately (code ${code})`));
257 + });
258 + });
259 + child.stdout?.destroy();
260 + child.stderr?.destroy();
261 + const port = Number(new URL(url).port);
262 + debugLog(`launched ${bin} port=${port}`);
263 + return {
264 + child,
265 + port,
266 + profileDir,
267 + spoolDir,
268 + access: !!opts.access,
269 + sessionId: null,
270 + capabilities: null,
271 + ws: null,
272 + nextId: 1,
273 + pending: new Map(),
274 + waiters: [],
275 + events: [],
276 + ledger: { subscriptions: [], intercepts: [], preloadScripts: [] },
277 + };
280 +async function teardown(): Promise<void> {
281 + const b = browser;
282 + browser = null;
283 + lastById = new Map();
284 + if (!b) return;
285 + try {
286 + b.ws?.close();
287 + } catch {
288 + /* ignore */
290 + b.child.kill("SIGTERM");
291 + await rm(b.profileDir, { recursive: true, force: true }).catch(() => {});
292 + await rm(b.spoolDir, { recursive: true, force: true }).catch(() => {});
295 +// ── screencast destination ──────────────────────────────────────────────
297 +function configuredScreencastDir(cwd: string): string | undefined {
298 + const sm = SettingsManager.create(cwd);
299 + const project = parseScreencastSettings(
300 + (sm.getProjectSettings() as Record<string, unknown>)["firefox-bidi"],
301 + );
302 + const global = parseScreencastSettings(
303 + (sm.getGlobalSettings() as Record<string, unknown>)["firefox-bidi"],
304 + );
305 + return project ?? global;
308 +// ── extension ───────────────────────────────────────────────────────────────
310 +export default function firefoxBidiExtension(pi: ExtensionAPI) {
311 + const activate = () => {
312 + const tools = pi.getActiveTools();
313 + if (!tools.includes("firefox_bidi"))
314 + pi.setActiveTools([...tools, "firefox_bidi"]);
315 + };
317 + pi.registerCommand("firefox", {
318 + description:
319 + "Activate the firefox_bidi tool (/firefox), or tear down (/firefox close)",
320 + getArgumentCompletions: (prefix: string): AutocompleteItem[] | null => {
321 + const filtered = [{ value: "close", label: "close" }].filter((i) =>
322 + i.value.startsWith(prefix),
323 + );
324 + return filtered.length > 0 ? filtered : null;
325 + },
326 + handler: async (args, ctx) => {
327 + if (args.trim() === "close") {
328 + await teardown();
329 + pi.setActiveTools(
330 + pi.getActiveTools().filter((t) => t !== "firefox_bidi"),
331 + );
332 + ctx.ui.notify("Firefox stopped, tool deactivated", "info");
333 + return;
335 + activate();
336 + ctx.ui.notify(
337 + "firefox_bidi activated — load skill firefox-bidi before composing frames",
338 + "info",
339 + );
340 + },
341 + });
343 + pi.registerTool({
344 + name: "firefox_bidi",
345 + label: "Firefox BiDi",
346 + description:
347 + "Drive Firefox with raw WebDriver BiDi frames (W3C protocol). " +
348 + "Send a batch of {id?, method, params} frames; get verbatim responses plus captured events. " +
349 + "Load the firefox-bidi skill before composing frames.",
350 + promptGuidelines: [
351 + "Batch related commands into one call; include session.new with capabilities on first use.",
352 + 'Reference earlier results inside later frames with {{id.result.path}} placeholders, e.g. navigate params context: "{{2.result.context}}".',
353 + ],
354 + parameters: Type.Object({
355 + frames: Type.Array(
356 + Type.Object(
358 + id: Type.Optional(
359 + Type.Integer({
360 + minimum: 1,
361 + description: "unique positive int; auto-assigned when omitted",
362 + }),
363 + ),
364 + method: Type.String({
365 + description: "module.command, e.g. browsingContext.navigate",
366 + }),
367 + params: Type.Optional(
368 + Type.Record(Type.String(), Type.Unknown(), {
369 + description: "command params object",
370 + }),
371 + ),
372 + },
373 + { additionalProperties: false },
374 + ),
375 + { description: "raw BiDi frames, executed sequentially" },
376 + ),
377 + wait: Type.Optional(
378 + Type.String({
379 + description:
380 + "block until this event method arrives (e.g. browsingContext.load)",
381 + }),
382 + ),
383 + waitTimeoutMs: Type.Optional(Type.Integer({ minimum: 1, default: 5000 })),
384 + frameTimeoutMs: Type.Optional(
385 + Type.Integer({ minimum: 1, default: 30000 }),
386 + ),
387 + launch: Type.Optional(
388 + Type.Object(
390 + headed: Type.Optional(
391 + Type.Boolean({ default: false, description: "default headless" }),
392 + ),
393 + access: Type.Optional(
394 + Type.Boolean({
395 + default: false,
396 + description:
397 + "privileged system realm; implies fresh temp profile",
398 + }),
399 + ),
400 + prefs: Type.Optional(
401 + Type.Array(Type.String(), {
402 + description: "pref=value entries, values parsed as JSON",
403 + }),
404 + ),
405 + },
406 + { additionalProperties: false },
407 + ),
408 + ),
409 + }),
410 + async execute(_id, params, _signal, _onUpdate, ctx) {
411 + try {
412 + await resolveScreencastDestinations(params.frames, () =>
413 + configuredScreencastDir(ctx.cwd),
414 + );
415 + debugLog(`cwd=${ctx.cwd} cfg=${configuredScreencastDir(ctx.cwd)}`);
416 + if (params.launch?.access && browser?.access === false)
417 + await teardown();
418 + if (!browser?.child.pid) {
419 + if (browser) await teardown();
420 + browser = await launch({
421 + headed: params.launch?.headed,
422 + access: params.launch?.access,
423 + prefs: params.launch?.prefs,
424 + });
426 + const b = browser;
427 + const ws = await sessionSocket(b);
429 + // validate + assign ids
430 + const seen = new Set<number>();
431 + params.frames.forEach((f) => {
432 + if (f.id === undefined) f.id = b.nextId++;
433 + if (seen.has(f.id)) throw new Error(`duplicate frame id ${f.id}`);
434 + seen.add(f.id);
435 + b.nextId = Math.max(b.nextId, f.id + 1);
436 + });
438 + const timeoutMs = params.frameTimeoutMs ?? DEFAULT_FRAME_TIMEOUT;
439 + const byId = new Map(lastById); // seed with previous batch: cross-call refs work
440 + const { responses, texts } = await runBatch(
441 + ws,
442 + b,
443 + // ids were assigned above; the optional tool schema cannot express that
444 + params.frames as Frame[],
445 + byId,
446 + timeoutMs,
447 + );
449 + const sn = responses.find((r) => r.method === "session.new");
450 + if (
451 + sn?.type === "success" &&
452 + (sn.result as Record<string, unknown>)?.sessionId
453 + ) {
454 + const r = sn.result as Record<string, unknown>;
455 + b.sessionId = r.sessionId as string;
456 + b.capabilities = r.capabilities as Record<string, unknown>;
459 + let wait: BidiEvent | { timedOut: true; method: string } | undefined;
460 + if (params.wait) {
461 + const method = params.wait;
462 + const buffered = b.events.find((e) => e.method === method);
463 + const hit =
464 + buffered ??
465 + (await new Promise<BidiEvent | null>((resolve) => {
466 + const t = setTimeout(
467 + () => resolve(null),
468 + params.waitTimeoutMs ?? 5000,
469 + );
470 + b.waiters.push({
471 + test: (e) => e.method === method,
472 + resolve: (f) => {
473 + clearTimeout(t);
474 + resolve(f);
475 + },
476 + });
477 + }));
478 + wait = hit ?? { timedOut: true, method };
481 + const events = b.events.slice(-EVENT_CAP);
482 + lastById = byId;
483 + const text = assemblePayload(
485 + sessionId: b.sessionId,
486 + port: b.port,
487 + capabilities: b.capabilities,
488 + },
489 + texts,
490 + events,
491 + EVENT_CAP,
492 + params.wait ? wait : undefined,
493 + );
494 + return {
495 + content: [{ type: "text", text }],
496 + details: {
497 + sessionId: b.sessionId,
498 + responses: responses.length,
499 + events: events.length,
500 + },
501 + };
502 + } catch (e) {
503 + throw new Error(e instanceof Error ? e.message : String(e));
505 + },
506 + });
508 + pi.on("resources_discover", () => ({ skillPaths: [SKILL_PATH] }));
510 + pi.on("session_start", (_event, ctx) => {
511 + // tool ships registered but inactive: /firefox (or the PI_FIREFOX=1
512 + // dev/testing hook) activates it, so the initial prompt stays free of
513 + // the tool contract
514 + if (process.env.PI_FIREFOX === "1") activate();
515 + else
516 + pi.setActiveTools(
517 + pi.getActiveTools().filter((t) => t !== "firefox_bidi"),
518 + );
519 + if (!process.env.FIREFOX_BIN && !resolveOnPath("firefox")) {
520 + ctx.ui.notify(
521 + "firefox-bidi: firefox not on PATH; set FIREFOX_BIN or install Firefox before /firefox",
522 + "warning",
523 + );
525 + });
527 + pi.on("session_shutdown", async () => {
528 + await teardown();
529 + });

extensions/firefox-bidi/lib.ts

Added
new file mode 100644
1 +import { mkdir, writeFile } from "node:fs/promises";
2 +import { join } from "node:path";
4 +export interface BidiResponse {
5 + id: number;
6 + method?: string;
7 + type?: string;
8 + result?: unknown;
9 + error?: string;
10 + message?: string;
11 + skipped?: boolean;
14 +export interface Frame {
15 + id: number;
16 + method: string;
17 + params?: Record<string, unknown>;
20 +export interface BatchCtx {
21 + pending: Map<number, (r: BidiResponse) => void>;
22 + ledger: {
23 + subscriptions: string[];
24 + intercepts: string[];
25 + preloadScripts: string[];
26 + };
27 + spoolDir: string;
30 +const MAX_INLINE = 256 * 1024;
32 +export function ledgerUpdate(
33 + b: Pick<BatchCtx, "ledger">,
34 + method: string,
35 + params: Record<string, unknown>,
36 + res: BidiResponse,
37 +): void {
38 + const ok = res.type === "success";
39 + const result = (res.result ?? {}) as Record<string, unknown>;
40 + if (method === "session.subscribe" && ok) {
41 + for (const e of (params.events as string[]) ?? [])
42 + if (!b.ledger.subscriptions.includes(e)) b.ledger.subscriptions.push(e);
43 + } else if (method === "session.unsubscribe") {
44 + const gone = new Set((params.events as string[]) ?? []);
45 + b.ledger.subscriptions = b.ledger.subscriptions.filter((x) => !gone.has(x));
46 + } else if (
47 + method === "network.addIntercept" &&
48 + ok &&
49 + typeof result.intercept === "string"
50 + ) {
51 + b.ledger.intercepts.push(result.intercept);
52 + } else if (method === "network.removeIntercept") {
53 + const ids = new Set(
54 + ((params.intercepts as string[]) ?? []).concat(
55 + (params.intercept as string[]) ?? [],
56 + ),
57 + );
58 + b.ledger.intercepts = b.ledger.intercepts.filter((x) => !ids.has(x));
59 + } else if (
60 + method === "script.addPreloadScript" &&
61 + ok &&
62 + typeof result.id === "string"
63 + ) {
64 + b.ledger.preloadScripts.push(result.id);
65 + } else if (
66 + method === "script.removePreloadScript" &&
67 + typeof params.id === "string"
68 + ) {
69 + b.ledger.preloadScripts = b.ledger.preloadScripts.filter(
70 + (x) => x !== params.id,
71 + );
72 + }
75 +// Serialize each response exactly once: the size check and the payload text
76 +// share one stringify. Oversized results are written to disk and replaced by
77 +// a small marker before serialization. Returns responses plus their JSON
78 +// texts so callers can assemble the payload without re-serializing.
79 +export async function runBatch(
80 + ws: Pick<WebSocket, "send">,
81 + b: BatchCtx,
82 + frames: Frame[],
83 + byId: Map<number, BidiResponse>,
84 + timeoutMs: number,
85 +): Promise<{ responses: BidiResponse[]; texts: string[] }> {
86 + const responses: BidiResponse[] = [];
87 + const texts: string[] = [];
88 + const spoolWrites: Promise<void>[] = [];
89 + let aborted = false;
90 + for (const frame of frames) {
91 + if (aborted) {
92 + const r = {
93 + id: frame.id,
94 + method: frame.method,
95 + skipped: true,
96 + } as BidiResponse;
97 + responses.push(r);
98 + texts.push(JSON.stringify(r));
99 + continue;
101 + let wire: unknown;
102 + try {
103 + wire = {
104 + id: frame.id,
105 + method: frame.method,
106 + params: interpolate(frame.params ?? {}, byId),
107 + };
108 + } catch (e) {
109 + const r: BidiResponse = {
110 + id: frame.id,
111 + method: frame.method,
112 + type: "error",
113 + error: "pi interpolation",
114 + message: e instanceof Error ? e.message : String(e),
115 + };
116 + responses.push(r);
117 + texts.push(JSON.stringify(r));
118 + aborted = true;
119 + continue;
121 + const res = await new Promise<BidiResponse>((resolve) => {
122 + const t = setTimeout(
123 + () =>
124 + resolve({
125 + id: frame.id,
126 + type: "timeout",
127 + message: `no response within ${timeoutMs}ms; side effects may have applied`,
128 + }),
129 + timeoutMs,
130 + );
131 + b.pending.set(frame.id, (r) => {
132 + clearTimeout(t);
133 + resolve(r);
134 + });
135 + ws.send(JSON.stringify(wire));
136 + });
137 + if (res.type === "timeout" || res.type === "connection-closed")
138 + aborted = true;
139 + ledgerUpdate(
140 + b,
141 + frame.method,
142 + (frame.params ?? {}) as Record<string, unknown>,
143 + res,
144 + );
145 + byId.set(frame.id, res);
146 + const out: BidiResponse = { ...res, id: frame.id, method: frame.method };
147 + let text = JSON.stringify(out);
148 + if (text.length > MAX_INLINE && res.type === "success") {
149 + const file = join(
150 + b.spoolDir,
151 + `${frame.method.replace(/[^\w.-]/g, "_")}-${frame.id}.json`,
152 + );
153 + const data = JSON.stringify(out.result ?? null);
154 + out.result = {
155 + spooled: file,
156 + size: text.length,
157 + note: "full result written to file; read selectively",
158 + } as unknown;
159 + text = JSON.stringify(out);
160 + // Write off the critical path: frames keep flowing while the file
161 + // flushes; the drain below completes before we return.
162 + spoolWrites.push(writeFile(file, data));
164 + responses.push(out);
165 + texts.push(text);
167 + await Promise.all(spoolWrites);
168 + return { responses, texts };
171 +// Assemble the tool payload from pre-serialized response texts so large
172 +// results are never serialized twice. Key order matches the historical
173 +// object literal: session, responses, events, note?, wait?.
174 +export function assemblePayload(
175 + session: unknown,
176 + responseTexts: string[],
177 + events: unknown[],
178 + eventCap: number,
179 + wait?: unknown,
180 +): string {
181 + let s =
182 + '{"session":' +
183 + JSON.stringify(session) +
184 + ',"responses":[' +
185 + responseTexts.join(",") +
186 + '],"events":' +
187 + JSON.stringify(events);
188 + if (events.length === eventCap)
189 + s += ',"note":"event buffer capped at ' + eventCap + '"';
190 + if (wait !== undefined) s += ',"wait":' + JSON.stringify(wait);
191 + return s + "}";
194 +// Strings in frame params may reference earlier responses: "{{2.result.context}}".
195 +// Resolved after the referenced frame completes, before this frame is sent.
196 +// Strings without the marker skip the regex entirely.
197 +export function interpolate(
198 + value: unknown,
199 + byId: Map<number, BidiResponse>,
200 +): unknown {
201 + if (typeof value === "string") {
202 + if (!value.includes("{{")) return value;
203 + return value.replace(
204 + /\{\{(\d+)((?:\.\w+(?:\.\d+)*)+)\}\}/g,
205 + (_m, idStr, pathStr) => {
206 + const res = byId.get(Number(idStr));
207 + if (res?.type !== "success")
208 + throw new Error(
209 + `{{${idStr}${pathStr}}}: frame ${idStr} has no successful response`,
210 + );
211 + let node: unknown = res;
212 + for (const seg of pathStr.replace(/^\./, "").split(".")) {
213 + if (node === null || node === undefined)
214 + throw new Error(`{{${idStr}${pathStr}}}: path ends at ${seg}`);
215 + node = (node as Record<string, unknown>)[seg];
217 + if (node === null || node === undefined || typeof node === "object")
218 + throw new Error(
219 + `{{${idStr}${pathStr}}}: reference must resolve to a primitive`,
220 + );
221 + return String(node);
222 + },
223 + );
225 + if (Array.isArray(value)) return value.map((v) => interpolate(v, byId));
226 + if (value && typeof value === "object") {
227 + return Object.fromEntries(
228 + Object.entries(value as Record<string, unknown>).map(([k, v]) => [
229 + k,
230 + interpolate(v, byId),
231 + ]),
232 + );
234 + return value;
237 +export function parseScreencastSettings(value: unknown): string | undefined {
238 + if (typeof value !== "object" || value === null || Array.isArray(value))
239 + return undefined;
240 + const dir = (value as Record<string, unknown>).screencastDir;
241 + return typeof dir === "string" && dir.trim() ? dir : undefined;
244 +// Gecko requires destinationFolder to be an existing directory and silently
245 +// falls back to ~/Downloads otherwise. Resolve per-call param > configured dir
246 +// > Downloads default, and create the directory up front.
247 +export async function resolveScreencastDestinations(
248 + frames: { method: string; params?: Record<string, unknown> }[],
249 + configured?: () => string | undefined,
250 +): Promise<void> {
251 + for (const frame of frames) {
252 + if (frame.method !== "browsingContext.startScreencast") continue;
253 + frame.params ??= {};
254 + const dir =
255 + (typeof frame.params.destinationFolder === "string" &&
256 + frame.params.destinationFolder.trim()
257 + ? frame.params.destinationFolder
258 + : undefined) ?? configured?.();
259 + if (!dir) continue; // Gecko default: ~/Downloads
260 + await mkdir(dir, { recursive: true });
261 + frame.params.destinationFolder = dir;

extensions/firefox-bidi/mise.toml

Added
new file mode 100644
1 +[tasks.fmt]
2 +description = "Format firefox-bidi extension"
3 +depends = ["//:install"]
4 +run = [
5 + "biome check --write --config-path ../.. .",
6 + "rumdl fmt --config ../../.rumdl.toml .",
7 + "taplo format --config ../../.taplo.toml mise.toml",
8 +]
10 +[tasks.lint]
11 +description = "Lint firefox-bidi extension"
12 +depends = ["//:install"]
13 +run = [
14 + "biome ci --config-path ../.. .",
15 + "rumdl check --config ../../.rumdl.toml .",
16 + "taplo format --check --config ../../.taplo.toml mise.toml",
19 +[tasks.test]
20 +description = "Test firefox-bidi extension"
21 +depends = ["//:install"]
22 +run = [
23 + { task = "//:test", args = ["extensions/firefox-bidi"] },
26 +[tasks.check]
27 +description = "Check firefox-bidi extension"
28 +depends = ["//:install"]
29 +run = [
30 + { task = "//:build", args = ["firefox-bidi"] },
33 +[tasks.bench]
34 +description = "Microbenchmark the batch hot path (fake socket)"
35 +depends = ["//:install"]
36 +run = "node --no-warnings --experimental-strip-types __bench__/runBatch.bench.mts 30"
38 +[tasks.bench-e2e]
39 +description = "Benchmark batches against a real headless Firefox (spawns a browser)"
40 +depends = ["//:install"]
41 +usage = '''
42 +arg "[pairs]" default="12"
43 +'''
44 +run = "node --no-warnings --experimental-strip-types __bench__/e2e.mts {{ usage.pairs }}"

extensions/firefox-bidi/package.json

Added
new file mode 100644
1 +{
2 + "name": "@bugabinga/pi-ext-firefox-bidi",
3 + "private": true,
4 + "type": "module",
5 + "main": "index.ts",
6 + "license": "MIT",
7 + "description": "Drive Firefox with raw WebDriver BiDi frames from Pi.",
8 + "keywords": [
9 + "pi",
10 + "pi-extension",
11 + "firefox",
12 + "webdriver-bidi"
13 + ],
14 + "peerDependencies": {
15 + "@earendil-works/pi-coding-agent": "*",
16 + "@earendil-works/pi-tui": "*",
17 + "typebox": "*"
18 + },
19 + "devDependencies": {
20 + "@marcfargas/pi-test-harness": "^0.6.1",
21 + "vitest": "^4.1.9"
22 + },
23 + "pi": {
24 + "extensions": [
25 + "index.ts"
26 + ]
27 + }

extensions/firefox-bidi/skills/firefox-bidi/SKILL.md

Added
new file mode 100644
1 +---
2 +name: firefox-bidi
3 +description: Drive Firefox via raw WebDriver BiDi JSON (websocket protocol). Use when automating, testing, scraping, debugging or screenshotting web pages in Firefox, when the user asks to control Firefox or a browser, or when composing browsingContext/script/network/storage/input BiDi frames for the firefox_bidi tool.
4 +---
6 +# Firefox via WebDriver BiDi
8 +Drive Firefox with raw W3C WebDriver BiDi frames through the `firefox_bidi`
9 +tool. You already know the protocol (module.command JSON frames). This skill
10 +adds: the tool contract, Firefox launch policy, dialect traps, and Firefox
11 +capability deltas.
13 +**Snapshot warning:** the ✗-tables below were verified against mozilla-central
14 +(≈ FF 157) and Firefox 156 release. Newer Firefox may support more. Always
15 +trust the error frame over the table: `unknown command` / `unsupported
16 +operation` = missing; anything else = try it. First `session.new` response
17 +includes `capabilities.browserVersion` — use it to judge which deltas apply.
19 +Tool availability: it loads on demand via the `/firefox` command. If a
20 +`firefox_bidi` tool call fails with "unknown tool", ask the user to run
21 +`/firefox`, then retry.
23 +## Tool contract
25 +```
26 +firefox_bidi({
27 + frames: [ // raw W3C BiDi, verbatim
28 + {"method":"session.new","params":{"capabilities":{"alwaysMatch":{"acceptInsecureCerts":true}}}},
29 + {"method":"browsingContext.create","params":{"type":"tab"}},
30 + {"method":"browsingContext.navigate","params":{"context":"<ctx>","url":"https://…","wait":"complete"}},
31 + {"method":"script.evaluate","params":{"expression":"document.title","target":{"context":"<ctx>"}}}
32 + ],
33 + wait: "network.responseCompleted", // optional: block for event after batch
34 + waitTimeoutMs: 5000, frameTimeoutMs: 30000,
35 + launch: {headed:false, access:false, prefs:["fprivacy.trackingprotection.enabled=false"]} // only when needed
36 +})
37 +→ {session:{sessionId,port,capabilities}, responses:[verbatim BiDi frames], events:[captured]}
38 +```
40 +Rules:
42 +- Batch related commands in one call; frames run sequentially, continue-on-error.
43 +- **Interpolation**: strings in later frames may reference earlier results with
44 + `{{<id>.result.path}}`, e.g. after frame 2 = `browsingContext.create` use
45 + `"context": "{{2.result.context}}"` in frame 3. Resolves to the response
46 + value at send time; unresolved refs abort the batch with a clear error.
47 + → One call can chain: create → navigate → evaluate → screenshot.
48 +- `id` optional (auto-assigned); must be unique positive ints when given.
49 +- First call of a task: start with `session.new` to control capabilities.
50 +- The browser persists between calls (one per session). `/firefox close` stops it.
51 +- Bulky results (screenshots, PDFs, bodies) are spooled to a file path in-band.
52 +- Event buffer: last 200 events captured continuously between calls, returned
53 + each call → console errors, downloads, auth prompts arriving between calls
54 + are visible in the next response.
55 +- Intercepts/preloads/subscriptions are tracked in a ledger; they persist until
56 + removed or `/firefox close`.
58 +## Dialect traps
60 +- `navigate`/`reload` `wait` values: `"none" | "interactive" | "complete"`.
61 + **Default `"none"`** → returns immediately. Classic-WebDriver `"load"` does
62 + not exist here. Prefer explicit `"wait":"complete"`.
63 +- Events need `session.subscribe` first. Subscriptions are session-global and
64 + module-expandable: `{"events":["network"]}` subscribes all network events.
65 +- Iframe/realm ids churn per navigation: re-`getTree` / re-`getRealms` after
66 + navigating. On `no such browsing context` / `no such node`: re-locate, never
67 + reuse stale ids.
68 +- Intercepts block matching requests **silently**: resolve or remove in the
69 + same batch, or all later navigations stall.
70 +- Headless has no OS window: `browser.setClientWindowState` is meaningless →
71 + use `browsingContext.setViewport`.
72 +- Screencast files: per-call `destinationFolder` (dir auto-created) →
73 + `firefox-bidi.screencastDir` in project or user `settings.json` →
74 + `~/Downloads`. Recordings are headless-safe. **`destinationFolder` needs
75 + Firefox 157+/nightly**; Firefox 156 ignores it and always writes to
76 + `~/Downloads`.
77 +- `script.evaluate` requires explicit `target: {context}` and **explicit
78 + `awaitPromise: true|false`** (Firefox asserts its presence; spec default
79 + false). Results come serialized (`sharedId`/handles); pass
80 + `awaitPromise:true` for promises.
82 +## Anti-hallucination (Firefox deltas vs living spec)
84 +| ✗ Not in Firefox | Use instead |
85 +|---|---|
86 +| `storage.getLocalStorageItems` / sessionStorage cmds | `script.evaluate("JSON.stringify(localStorage)")` |
87 +| `browsingContext.setBypassCSP` | preload scripts respect CSP; no bypass |
88 +| emulation: forcedColors/mediaFeatures/scripting/touch/viewportMeta overrides | only 7 exist: geolocation, locale, networkConditions, screenOrientation, screenSettings, timezone, userAgent |
89 +| `browsingContext.navigationAborted` event | navigationStarted/Committed/Failed only |
90 +| chrome-scope / UI automation | no BiDi module; OS tools instead |
91 +| CDP: metrics, coverage, heap | `moz.profiler.start/stop`, `moz.debugging` |
93 +Full matrix: `references/feature-matrix.md` — read before exotic commands.
95 +## Launch policy
97 +- Always fresh temp profile (automatic). Never the user's profile.
98 +- `launch.access:true` = `--remote-allow-system-access`: script.evaluate gains
99 + privileged scope (`Services.prefs` for runtime prefs). Treat system-realm
100 + script text like shell commands.
101 +- Default headless. This includes `startScreencast` video recording: works
102 + headless, produces a valid webm. Use `launch.headed:true` only when the human
103 + wants to watch the browser live, never for artifacts.
104 +- `launch.prefs` → `user.js` before start (values parsed as JSON:
105 + `key=true`, `key=42`, `key="string"`).
107 +## Patterns
109 +Extract after navigation: `navigate(wait:"complete")` → `script.evaluate`.
110 +Element interaction: `browsingContext.locateNodes` (css/xpath/innerText/
111 +accessibility/context locators) → use `sharedId`/`nodeRef` in
112 +`input.performActions` or `script.callFunction`.
113 +Network capture: `session.subscribe ["network"]` → act →
114 +`wait:"network.responseCompleted"` → `network.getData`.
115 +Console tail: returned `events` each call; filter `log.entryAdded`.
117 +## Failure recovery
119 +- `connect failed` / firefox exited → stale or crashed browser: retry with
120 + `launch` (fresh browser, new context ids).
121 +- `Maximum number of active sessions` → orphan session in a live browser;
122 + `/firefox close`, then retry.
123 +- Frame `timeout` → side effects may have applied; re-check state with
124 + `browsingContext.getTree` before retrying.

extensions/firefox-bidi/skills/firefox-bidi/references/feature-matrix.md

Added
new file mode 100644
1 +<!-- GENERATED COPY: source = research/firefox-feature-matrix.md in the
2 + prototype repo ~/Workspace/firefox-bidi. Regenerate there, then re-copy.
3 + Snapshot date and sources are documented in the source file. -->
5 +# Firefox features: available via WebDriver BiDi vs not
7 +Snapshot of this gecko checkout (recent nightly), verified against source paths below.
8 +Cross-reference: command dispatch and client API notes live in the prototype repo (`~/Workspace/firefox-bidi/research/modules/`).
9 +Feature-detect at runtime: unsupported → `unknown command` / `unsupported operation` error.
11 +## ✅ Available via BiDi
13 +### Session & capabilities
15 +| Feature | Commands / evidence |
16 +|---|---|
17 +| Session lifecycle | `session.new` (HTTP `POST /session` or on WS), `session.status`, `session.end` — `remote/webdriver-bidi/WebDriverBiDiConnection.sys.mjs:202-227`, `root/session.sys.mjs:80` |
18 +| Event subscriptions, granular | `session.subscribe`/`unsubscribe` with `contexts` (all / top-level browsing contexts / user contexts) — `root/session.sys.mjs:118,234` |
19 +| Capabilities | `acceptInsecureCerts`, `proxy` (pref-backed: direct/pac/system/manual), `pageLoadStrategy`, `strictFileInteractability`, `timeouts`, `unhandledPromptBehavior`, `webSocketUrl` — `remote/shared/webdriver/Capabilities.sys.mjs:33-36,480-486,592,748` |
20 +| Firefox caps | `moz:firefoxOptions` (args + prefs at launch), `moz:accessibilityChecks`, `moz:webdriverClick`, `moz:processID`, `moz:profile` — `Capabilities.sys.mjs:39-41,493-494` |
22 +### Contexts & navigation
24 +| Feature | Commands / evidence |
25 +|---|---|
26 +| Open/close/inspect contexts | `browsingContext.create` (tab or window), `close`, `getTree` (full iframe tree) — `root/browsingContext.sys.mjs:660,605,946` |
27 +| Navigate/reload/history | `navigate` (wait states), `reload`, `traverseHistory`, `activate` — `browsingContext.sys.mjs:1400,1656,2231,333` |
28 +| Viewport | `browsingContext.setViewport` (viewport + devicePixelRatio) — `browsingContext.sys.mjs:1748` |
29 +| Navigation events | `contextCreated/Destroyed`, `navigationStarted/Committed/Failed`, `fragmentNavigated`, `domContentLoaded`, `load`, `historyUpdated` — `browsingContext.sys.mjs` `supportedEvents` |
30 +| Window management | `browser.getClientWindows`, `browser.setClientWindowState` (normal/maximized/minimized/fullscreen + x/y/width/height) — `root/browser.sys.mjs:205,478`, states `:77-80` |
31 +| User contexts (containers) | `browser.createUserContext`, `getUserContexts`, `removeUserContext` — `browser.sys.mjs:240,291,421` |
33 +### Scripting & DOM
35 +| Feature | Commands / evidence |
36 +|---|---|
37 +| JS evaluation | `script.evaluate`, `script.callFunction` — serialization (sharedId/handles, ownership modes), sandbox realms, channels (async result push) — `root/script.sys.mjs:421,556` |
38 +| Preload scripts | `addPreloadScript`/`removePreloadScript` (document-start, args) — `script.sys.mjs:167,733` |
39 +| Realms | `script.getRealms`: `window`, `dedicated-worker`, `shared-worker`, `service-worker` — `script.sys.mjs:678` + unsupported list (audio-worklet/paint-worklet/worker/worklet); events `realmCreated/Destroyed`, `script.message` |
40 +| Element location | `browsingContext.locateNodes`: `css`, `xpath`, `innerText`, `accessibility`, `context` (into iframes); near-miss reporting — `browsingContext.sys.mjs:1118-1167,1224` |
41 +| Screenshots / PDF | `captureScreenshot` (viewport/document origin, clip), `print` (PDF) — `browsingContext.sys.mjs:440,1555` |
43 +### Input
45 +| Feature | Commands / evidence |
46 +|---|---|
47 +| Actions | `input.performActions`, `releaseActions` (keyboard, pointer, wheel; per spec input sources) — `root/input.sys.mjs:293,334` |
48 +| File upload | `input.setFiles` + `input.fileDialogOpened` event — `input.sys.mjs:382,410-411` |
50 +### Network
52 +| Feature | Commands / evidence |
53 +|---|---|
54 +| Observe | events `beforeRequestSent`, `responseStarted`, `responseCompleted`, `fetchError`, `authRequired` — `root/network.sys.mjs` `supportedEvents` |
55 +| Intercept/modify | `addIntercept`, `continueRequest`, `continueResponse`, `continueWithAuth`, `failRequest`, `provideResponse`, `removeIntercept` — `network.sys.mjs:400-1117` |
56 +| Cache + capture | `setCacheBehavior`; data collectors `addDataCollector`/`getData`/`disownData`/`removeDataCollector` (response body capture) — `network.sys.mjs` |
58 +### Storage, permissions, extensions, emulation, misc
60 +| Feature | Commands / evidence |
61 +|---|---|
62 +| Cookies (partitioned) | `storage.getCookies`/`setCookie`/`deleteCookies` with `partition` (incl. user contexts) — `root/storage.sys.mjs:157,199,258` |
63 +| Permissions | `permissions.setPermission` (granted/denied, per origin + user context) — `root/permissions.sys.mjs:54-114` |
64 +| Extensions | `webExtension.install` (`moz:allowPrivateBrowsing`, `moz:permanent`), `uninstall`, `moz:listExtensions` — `root/webExtension.sys.mjs:141,237,263` |
65 +| Emulation (7) | `emulation.setGeolocationOverride`, `setLocaleOverride`, `setNetworkConditions` (throttling), `setScreenOrientationOverride`, `setScreenSettingsOverride`, `setTimezoneOverride`, `setUserAgentOverride` — `root/emulation.sys.mjs:137-631`; windowglobal variants → per-context |
66 +| Prompts | `browsingContext.handleUserPrompt` + `userPromptOpened/Closed` events — `browsingContext.sys.mjs:1039` |
67 +| Downloads | `browser.setDownloadBehavior` (global or per user context, null resets) + `downloadWillBegin`/`downloadEnd` events — `browser.sys.mjs:317`, BC `supportedEvents` |
68 +| Console logging | `log.entryAdded` (console API + JS errors with stack traces) — `modules/log.md` |
69 +| Screencast (recording) | `browsingContext.startScreencast`/`stopScreencast` → webm file on disk (no audio, no frame streaming) — `browsingContext.sys.mjs:1968,2187` |
71 +### Firefox-only extensions (`moz:` namespace)
73 +| Feature | Commands / evidence |
74 +|---|---|
75 +| JS debugger | `moz.debugging.setDebuggerEnabled`, `setBreakpoint`, `removeBreakpoint`, `resume`, `stepOver/Into/Out`, `listScripts`, `getScriptSource` — `root/moz/debugging.sys.mjs:77-346` |
76 +| Gecko profiler | `moz.profiler.start`/`stop` — `root/moz/profiler.sys.mjs:60,206` |
78 +## ⚠️ Partial / caveated
80 +- **Worker realms**: listable (`getRealms` accepts service/dedicated/shared worker types) but worklets unsupported; evaluation *inside* worker realms not verified in this checkout — UNVERIFIED.
81 +- **Screencast**: file-based webm recording only; no per-frame `screencastFrame` event streaming, no audio track — `browsingContext.sys.mjs:1927-1966`.
82 +- **Downloads**: behavior config + events only; BiDi never hands you the file, `setDownloadBehavior` chooses folder.
83 +- **Proxy**: session-start only, applies profile-wide via prefs — no per-context proxy.
84 +- **Permissions**: set-only; no read/query command.
85 +- **Emulation resets**: per-command `null` reset semantics vary; no clear-all command.
86 +- **Multiple sessions**: many sessions per browser OK, but each WS serves exactly one session (`wire-protocol.md`); no event broadcast across sessions.
88 +## ❌ Not available via BiDi (as of this checkout)
90 +| Missing feature | Evidence / workaround |
91 +|---|---|
92 +| localStorage / sessionStorage inspection & mutation (spec'd `storage.*` item commands) | `root/storage.sys.mjs` implements cookies only → workaround: `script.evaluate("localStorage…")` |
93 +| `browsingContext.setBypassCSP` | absent from `root/browsingContext.sys.mjs` → preload scripts can't violate CSP either |
94 +| `browsingContext.navigationAborted` event | not in BC `supportedEvents` |
95 +| Abort/redirect a navigation request proactively | interception reacts to network layer; no navigation-level cancel command |
96 +| Pre-provisioned HTTP auth credentials | must react to `network.authRequired` with `continueWithAuth` — no credential store command |
97 +| Browser chrome/UI automation (menus, toolbars, DevTools UI, profile manager) | no chrome-scope BiDi modules (`ModuleRegistry`: webcontent destinations only). Escape hatch: `--remote-allow-system-access` + script evaluation in privileged scope — powerful, unsupported for normal automation |
98 +| Runtime prefs management | prefs only at launch (`moz:firefoxOptions.prefs`); no prefs module |
99 +| Full accessibility-tree snapshot (spec proposal) | only `locateNodes` with `accessibility` locator |
100 +| Performance metrics / JS coverage / heap snapshots (CDP-parity) | no BiDi module; `moz.profiler` is the only adjacent tool |
101 +| Clipboard read/write, notifications permission prompts beyond `setPermission` | no commands |
102 +| Device/mobile emulation combos (touch forcing, device descriptors) | only UA + viewport + orientation + locale + timezone + networkConditions |
103 +| Remote (non-loopback) exposure | listener binds localhost by design; no auth/TLS — tunnel via SSH |
104 +| BiDi control of the Firefox UI window chrome / OS dialogs | OS-level; only `input.fileDialogOpened` interception for file pickers |
105 +| Gamepad, geolocation *watch* mocking beyond override, WebRTC fakery | no commands |
107 +## Quick check recipe
109 +Ground truth for "does Firefox X support Y": `gecko/remote/webdriver-bidi/modules/ModuleRegistry.sys.mjs` (module list) + grep the root module for `async <command>(` and `supportedEvents`. Missing there = not available, regardless of spec.

package-lock.json

Modified
126 126 "@earendil-works/pi-coding-agent": "*"
129 + "extensions/firefox-bidi": {
130 + "name": "@bugabinga/pi-ext-firefox-bidi",
131 + "license": "MIT",
132 + "devDependencies": {
133 + "@marcfargas/pi-test-harness": "^0.6.1",
134 + "vitest": "^4.1.9"
135 + },
136 + "peerDependencies": {
137 + "@earendil-works/pi-coding-agent": "*",
138 + "@earendil-works/pi-tui": "*",
139 + "typebox": "*"
141 + },
129 142 "extensions/footer": {
130 143 "name": "@bugabinga/pi-ext-footer",
131 144 "license": "MIT",
1303 1316 "resolved": "extensions/fast",
1304 1317 "link": true
1319 + "node_modules/@bugabinga/pi-ext-firefox-bidi": {
1320 + "resolved": "extensions/firefox-bidi",
1321 + "link": true
1322 + },
1306 1323 "node_modules/@bugabinga/pi-ext-footer": {
1307 1324 "resolved": "extensions/footer",
1308 1325 "link": true