Luigit
repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by admin

extensions/nushell/E2E.md

Raw
Rendered preview

Nushell JSON-mode evidence

//extensions/nushell:e2e-json invokes real Pi with --print --mode json --no-session --nushell, an isolated cwd, explicit Nu tool allowlist, and opt-in JSONL diagnostics. Each case retains invocation.json, events.jsonl, debug.jsonl, analysis.json, result.json, final.txt, stderr.txt, and generated modules/fixtures under <OS temp>/pi-ext-nushell-json/<run>/ in a private directory. The trace case also retains validation.json, which compares its generated Nu module against independent JS calculations over completed real debug logs. --verify <run-directory> repeats only trace validation for a retained run.

Observations

The first restricted invocation at .pi/tmp/nushell-json/2026-09-26T18-18-39-460Z/ exposed a Pi CLI allowlist conflict: --tools bash,read,write,edit excluded both Nu tools despite --nushell. Activation now checks Pi's configured registry before displacing shell tools and reports excluded tools instead of claiming success. An unrestricted pilot at .pi/tmp/nushell-json/2026-09-26T18-19-45-271Z/ demonstrated a model invoking ^nu -c inside nu, adding an avoidable second process. Tool guidance now directs agents to pass Nu source directly, avoid nested Nu processes, and prefer Nu builtins for suitable work.

All five module exercises plus the session-state probe in .pi/tmp/nushell-json/2026-09-26T18-43-52-847Z/ settled with Nu tool calls and no nested Nu invocations. That six-case run cost $0.4304 in projected API pricing and took 473 seconds; code-generation errors were corrected by agents before their final checks. The first generated trace.nu consumed an invented JSONL schema despite passing its own synthetic fixture, so these agent claims alone did not prove actual logger compatibility.

The current generated diagnostics correlate interleaved spans by spanId, pair starts with terminal events, and record the Pi process ID without command bodies or raw errors. Tests cover overlapping fresh calls, preflight failures, worker startup failures, private file permissions, error-code redaction, session state, and abort/reset handling. The trace and state probes in .pi/tmp/nushell-json/2026-09-26T19-02-22-798Z/ use real diagnostic fields and generated fixtures. The independent verifier initially failed because its stripped environment made Mise's already-configured file appear untrusted; trace/validation-failed-mise.json retains the exact failure. Restoring the existing Mise environment made verification run, then a harmless generated-result field naming difference caused a strict equality failure retained in trace/validation-schema-diff.json. Semantic normalization now compares the generated module's summaries with independent counts and nearest-rank percentiles over both completed real logs; trace/validation.json records two passes.

bench-current.txt measured 36.25 ms cold, 36.72 ms fresh, and 822 µs warm, 25 runs each. Later correlated-logger samples were 10–20% slower, but bench-load.txt documents substantial concurrent compiler load; no causal performance claim follows from that comparison.

Limits

These are bounded integration probes, not proof of zero bugs or universal agent optimality. The 110-second MCP deadline, detached external-process cleanup after cancellation, other operating systems, and diverse model behavior remain unverified live. Generated Nu modules are prototypes in evidence workspaces, not promoted packages. Do not read initial trace.nu prototypes as compatible with PI_NUSHELL_DEBUG; use the verified later trace fixture and module.

# Nushell JSON-mode evidence

`//extensions/nushell:e2e-json` invokes real Pi with `--print --mode json --no-session --nushell`, an isolated cwd, explicit Nu tool allowlist, and opt-in JSONL diagnostics.
Each case retains `invocation.json`, `events.jsonl`, `debug.jsonl`, `analysis.json`, `result.json`, `final.txt`, `stderr.txt`, and generated modules/fixtures under `<OS temp>/pi-ext-nushell-json/<run>/` in a private directory.
The trace case also retains `validation.json`, which compares its generated Nu module against independent JS calculations over completed **real** debug logs.
`--verify <run-directory>` repeats only trace validation for a retained run.

## Observations

The first restricted invocation at `.pi/tmp/nushell-json/2026-09-26T18-18-39-460Z/` exposed a Pi CLI allowlist conflict: `--tools bash,read,write,edit` excluded both Nu tools despite `--nushell`.
Activation now checks Pi's configured registry before displacing shell tools and reports excluded tools instead of claiming success.
An unrestricted pilot at `.pi/tmp/nushell-json/2026-09-26T18-19-45-271Z/` demonstrated a model invoking `^nu -c` inside `nu`, adding an avoidable second process.
Tool guidance now directs agents to pass Nu source directly, avoid nested Nu processes, and prefer Nu builtins for suitable work.

All five module exercises plus the session-state probe in `.pi/tmp/nushell-json/2026-09-26T18-43-52-847Z/` settled with Nu tool calls and no nested Nu invocations.
That six-case run cost $0.4304 in projected API pricing and took 473 seconds; code-generation errors were corrected by agents before their final checks.
The first generated `trace.nu` consumed an invented JSONL schema despite passing its own synthetic fixture, so these agent claims alone did **not** prove actual logger compatibility.

The current generated diagnostics correlate interleaved spans by `spanId`, pair starts with terminal events, and record the Pi process ID without command bodies or raw errors.
Tests cover overlapping fresh calls, preflight failures, worker startup failures, private file permissions, error-code redaction, session state, and abort/reset handling.
The trace and state probes in `.pi/tmp/nushell-json/2026-09-26T19-02-22-798Z/` use real diagnostic fields and generated fixtures.
The independent verifier initially failed because its stripped environment made Mise's already-configured file appear untrusted; `trace/validation-failed-mise.json` retains the exact failure.
Restoring the existing Mise environment made verification run, then a harmless generated-result field naming difference caused a strict equality failure retained in `trace/validation-schema-diff.json`.
Semantic normalization now compares the generated module's summaries with independent counts and nearest-rank percentiles over both completed real logs; `trace/validation.json` records two passes.

`bench-current.txt` measured 36.25 ms cold, 36.72 ms fresh, and 822 µs warm, 25 runs each.
Later correlated-logger samples were 10–20% slower, but `bench-load.txt` documents substantial concurrent compiler load; no causal performance claim follows from that comparison.

## Limits

These are bounded integration probes, not proof of zero bugs or universal agent optimality.
The 110-second MCP deadline, detached external-process cleanup after cancellation, other operating systems, and diverse model behavior remain unverified live.
Generated Nu modules are **prototypes in evidence workspaces**, not promoted packages.
Do not read initial `trace.nu` prototypes as compatible with `PI_NUSHELL_DEBUG`; use the verified later trace fixture and module.