Luigit
repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by admin

test/fixture/README.md

Raw
Rendered preview

Owned Pi test fixture

create-pi-session.ts builds a real Pi session through public APIs only (createAgentSession, DefaultResourceLoader, SessionManager.inMemory, ModelRuntime with a deterministic faux provider). It replaces the patched @marcfargas/pi-test-harness and reaches no private Pi internals.

API

createPiSession(options) returns a PiSession.

Options (PiSessionOptions):

  • extensionPaths?: string[] — absolute entrypoint paths to load (the path-loaded contract).
  • extensionFactories?: ExtensionFactory[] — in-process factories loaded in the test's own module graph. Use ONLY for component tests that must vi.mock/vi.spyOn an extension's own lazy-imported internal module (path-loaded extensions run under jiti in an isolated graph where such mocks do not apply). Keep a separate path-loaded test for the load/registration contract.
  • cwd?: string — working directory; a temp one is created and owned when omitted.
  • userSettings?: Record<string, unknown> — written to the isolated agent dir.
  • env?: Record<string, string | undefined> — overrides on top of the sandbox.
  • responses?: FauxResponseStep[] — scripted assistant turns.

PiSession:

  • session — the real AgentSession.
  • extensions — the loaded Extension[] from the public loader API. Use session.extensions[0].tools / .commands / .handlers / .path. Never read session.session.extensionRunner.extensions — that property is private and fails strict typechecking.
  • cwd, agentDir, env — the owned sandbox (env.XDG_STATE_HOME is cwd/.test-state).
  • errors: string[] — extension load/bind errors; assert [].
  • setResponses, providerState — scripted-turn controls.
  • dispose(): Promise<void> — shuts down and restores env; call in afterEach.

Migrating a harness.test.ts to contract.test.ts

Create extensions/<name>/__tests__/contract.test.ts reproducing every test and assertion from harness.test.ts, sourced from this fixture:

  • Import:

    import { fileURLToPath } from "node:url";
    import { afterEach, describe, expect, it } from "vitest";
    import {
    	createPiSession,
    	type PiSession,
    } from "../../../test/fixture/create-pi-session.ts";
    
  • Replace createTestSession({ extensions: [p] }) or createTestSession({ extensionFactories: [f] }) with:

    session = await createPiSession({
    	extensionPaths: [fileURLToPath(new URL("../index.ts", import.meta.url))],
    });
    

    Always path-load ../index.ts, even when the original used a factory.

  • Option mapping: harness env to fixture env; harness cwd to fixture cwd; harness-managed user settings to fixture userSettings. If the original wrote a project .pi/settings.json manually, keep doing that literally.

  • Accessor mapping: t.session to session.session; t.env to session.env; t.cwd to session.cwd. Name the variable session of type PiSession.

  • Dispose is async:

    let session: PiSession | undefined;
    afterEach(async () => {
    	await session?.dispose();
    	session = undefined;
    });
    

    Keep any vi.unstubAllEnvs() / vi.restoreAllMocks() calls.

  • Preserve all pure-unit tests (no session) verbatim.

  • Keep vi mocks/spies; they target session.session or globals.

  • The fixture binds in print mode and does not auto-emit lifecycle events. If a test depends on session_start / session_shutdown side effects, emit them explicitly, matching the original's intent:

    await session.session.extensionRunner.emit({
    	type: "session_start",
    	reason: "startup",
    });
    await session.session.extensionRunner.emit({
    	type: "session_shutdown",
    	reason: "quit",
    });
    
  • Do not weaken, skip, or delete assertions to make tests pass. Behavior must be preserved exactly.

Tests that mock an extension's internal module

If a harness test used extensionFactories: [ext] and vi.mock/vi.spyOn on a module the extension imports (e.g. node:fs/promises, ./search.js), reproduce it with createPiSession({ extensionFactories: [ext] }) (import the extension's default factory) so the mock and the extension share one module graph. Keep one additional path-loaded test (extensionPaths) for the load/registration contract.

Scripted model turns (playbook replacement)

The legacy harness run(says(...)) playbook maps to the fixture's scripted provider: createPiSession({ responses: [fauxAssistantMessage("..."), ...] }) (or session.setResponses([...])), then await session.session.prompt("..."). Use fauxAssistantMessage / fauxToolCall from @earendil-works/pi-ai. Assert observable results (tool executions, session messages via session.session.sessionManager) instead of reaching into stream internals.

settings.test.ts (settings resolution)

  • Project trust: session.session.settingsManager.setProjectTrusted(false).
  • User settings: write to path.join(session.agentDir, "settings.json") (the fixture sets PI_CODING_AGENT_DIR to agentDir).
  • Project settings: write to path.join(session.cwd, ".pi", "settings.json").
  • Per-test provider: session.session.extensionRunner.createContext().modelRegistry.registerProvider(faux.provider) then faux.setResponses([...]).
  • Captured warnings/notify: install a recording UI context via the runner and vi.spyOn its notify, replacing the harness t.events.uiCallsFor("notify").

debug.test.ts (debug trace)

  • Enable via the fixture env option: env: { PI_<EXT>_DEBUG: "1" }. The fixture applies env BEFORE loading, so the debug module reads it at import; withProcessEnv/vi.resetModules() are not needed.
  • Read records from path.join(session.env.XDG_STATE_HOME, "pi-ext", "debug", "<ext>").
  • Debug records flush asynchronously; poll (e.g. vi.waitFor or a deadline loop) until the expected events appear before asserting.
  • If the test must mock the extension's own imported module, load via extensionFactories (see above).

Do NOT modify files under __bench__/, __e2e__/, or any *.live.ts.

Then delete extensions/<name>/__tests__/harness.test.ts.

Verify: npx biome check --write extensions/<name>/__tests__/contract.test.ts then npm run test -- extensions/<name> and read the full result.

# Owned Pi test fixture

`create-pi-session.ts` builds a real Pi session through public APIs only
(`createAgentSession`, `DefaultResourceLoader`, `SessionManager.inMemory`,
`ModelRuntime` with a deterministic faux provider).
It replaces the patched `@marcfargas/pi-test-harness` and reaches no private Pi
internals.

## API

`createPiSession(options)` returns a `PiSession`.

Options (`PiSessionOptions`):

- `extensionPaths?: string[]` — absolute entrypoint paths to load (the path-loaded contract).
- `extensionFactories?: ExtensionFactory[]` — in-process factories loaded in the test's own module graph. Use ONLY for component tests that must `vi.mock`/`vi.spyOn` an extension's own lazy-imported internal module (path-loaded extensions run under jiti in an isolated graph where such mocks do not apply). Keep a separate path-loaded test for the load/registration contract.
- `cwd?: string` — working directory; a temp one is created and owned when omitted.
- `userSettings?: Record<string, unknown>` — written to the isolated agent dir.
- `env?: Record<string, string | undefined>` — overrides on top of the sandbox.
- `responses?: FauxResponseStep[]` — scripted assistant turns.

`PiSession`:

- `session` — the real `AgentSession`.
- `extensions` — the loaded `Extension[]` from the public loader API. Use
  `session.extensions[0].tools` / `.commands` / `.handlers` / `.path`. **Never**
  read `session.session.extensionRunner.extensions` — that property is private
  and fails strict typechecking.
- `cwd`, `agentDir`, `env` — the owned sandbox (`env.XDG_STATE_HOME` is `cwd/.test-state`).
- `errors: string[]` — extension load/bind errors; assert `[]`.
- `setResponses`, `providerState` — scripted-turn controls.
- `dispose(): Promise<void>` — shuts down and restores env; call in `afterEach`.

## Migrating a `harness.test.ts` to `contract.test.ts`

Create `extensions/<name>/__tests__/contract.test.ts` reproducing every test and
assertion from `harness.test.ts`, sourced from this fixture:

- Import:

  ```ts
  import { fileURLToPath } from "node:url";
  import { afterEach, describe, expect, it } from "vitest";
  import {
  	createPiSession,
  	type PiSession,
  } from "../../../test/fixture/create-pi-session.ts";
  ```

- Replace `createTestSession({ extensions: [p] })` or
  `createTestSession({ extensionFactories: [f] })` with:

  ```ts
  session = await createPiSession({
  	extensionPaths: [fileURLToPath(new URL("../index.ts", import.meta.url))],
  });
  ```

  Always path-load `../index.ts`, even when the original used a factory.

- Option mapping: harness `env` to fixture `env`; harness `cwd` to fixture `cwd`;
  harness-managed user settings to fixture `userSettings`. If the original wrote a
  project `.pi/settings.json` manually, keep doing that literally.
- Accessor mapping: `t.session` to `session.session`; `t.env` to `session.env`;
  `t.cwd` to `session.cwd`. Name the variable `session` of type `PiSession`.
- Dispose is async:

  ```ts
  let session: PiSession | undefined;
  afterEach(async () => {
  	await session?.dispose();
  	session = undefined;
  });
  ```

  Keep any `vi.unstubAllEnvs()` / `vi.restoreAllMocks()` calls.

- Preserve all pure-unit tests (no session) verbatim.
- Keep `vi` mocks/spies; they target `session.session` or globals.
- The fixture binds in `print` mode and does not auto-emit lifecycle events. If a
  test depends on `session_start` / `session_shutdown` side effects, emit them
  explicitly, matching the original's intent:

  ```ts
  await session.session.extensionRunner.emit({
  	type: "session_start",
  	reason: "startup",
  });
  await session.session.extensionRunner.emit({
  	type: "session_shutdown",
  	reason: "quit",
  });
  ```

- Do not weaken, skip, or delete assertions to make tests pass. Behavior must be
  preserved exactly.

### Tests that mock an extension's internal module

If a harness test used `extensionFactories: [ext]` and `vi.mock`/`vi.spyOn` on a
module the extension imports (e.g. `node:fs/promises`, `./search.js`), reproduce
it with `createPiSession({ extensionFactories: [ext] })` (import the extension's
default factory) so the mock and the extension share one module graph. Keep one
additional path-loaded test (`extensionPaths`) for the load/registration contract.

### Scripted model turns (playbook replacement)

The legacy harness `run(says(...))` playbook maps to the fixture's scripted
provider: `createPiSession({ responses: [fauxAssistantMessage("..."), ...] })`
(or `session.setResponses([...])`), then `await session.session.prompt("...")`.
Use `fauxAssistantMessage` / `fauxToolCall` from `@earendil-works/pi-ai`. Assert
observable results (tool executions, session messages via
`session.session.sessionManager`) instead of reaching into stream internals.

### settings.test.ts (settings resolution)

- Project trust: `session.session.settingsManager.setProjectTrusted(false)`.
- User settings: write to `path.join(session.agentDir, "settings.json")` (the fixture sets `PI_CODING_AGENT_DIR` to `agentDir`).
- Project settings: write to `path.join(session.cwd, ".pi", "settings.json")`.
- Per-test provider: `session.session.extensionRunner.createContext().modelRegistry.registerProvider(faux.provider)` then `faux.setResponses([...])`.
- Captured warnings/notify: install a recording UI context via the runner and `vi.spyOn` its `notify`, replacing the harness `t.events.uiCallsFor("notify")`.

### debug.test.ts (debug trace)

- Enable via the fixture `env` option: `env: { PI_<EXT>_DEBUG: "1" }`. The fixture applies env BEFORE loading, so the debug module reads it at import; `withProcessEnv`/`vi.resetModules()` are not needed.
- Read records from `path.join(session.env.XDG_STATE_HOME, "pi-ext", "debug", "<ext>")`.
- Debug records flush asynchronously; poll (e.g. `vi.waitFor` or a deadline loop) until the expected events appear before asserting.
- If the test must mock the extension's own imported module, load via `extensionFactories` (see above).

Do NOT modify files under `__bench__/`, `__e2e__/`, or any `*.live.ts`.

Then delete `extensions/<name>/__tests__/harness.test.ts`.

Verify: `npx biome check --write extensions/<name>/__tests__/contract.test.ts`
then `npm run test -- extensions/<name>` and read the full result.