Luigit
repositories / pi-ext

pi-ext

bugabingas pi extensions

owned by admin

extensions/voice/TESTING.md

Raw
Rendered preview

Voice verification

Safety and scope

Preserve active human calls; stopping, reloading or replacing them requires approval. Use an isolated Pi session, harmless requests and approved model/cache/history locations. Keep temporary fixtures, traces, relocated artifacts and recordings in OS temp. Obtain approval before native builds, model downloads/loads, hardware access, diagnostic writes outside approved roots or user configuration changes. Run checks serially with at least 320 MiB available RAM and NODE_OPTIONS=--max-old-space-size=160, GOMEMLIMIT=96MiB, GOMAXPROCS=1, RAYON_NUM_THREADS=1. Budget native model RAM separately before loading any real speech stack. Clear obsolete native opt-ins; removed PI_VOICE_TEST_AUDIO_ENDPOINT and addon/model environment variables do not activate a Voice test mode. Declare expected recognition, reply meaning, timing bounds and permitted normalization before testing; never relax them after a failure.

Fast client checks

Use Mise tasks; no native build, model, hardware or download is needed for deterministic client checks.

  • //extensions/voice:test: conversation, controller, history, command/UI and RPC regressions.
  • //extensions/voice:e2e: fast fake-backed public command/RPC scenarios; excludes real-time replay.
  • //extensions/voice:e2e-realtime: retained real-time recorded-speech UI acceptance, selected explicitly.
  • //extensions/voice:check: typecheck, static package check and bundled Voice build.
  • //extensions/voice:verify-package: build and load the relocated bundled client outside the checkout.
  • //extensions/voice:lint: formatting, Markdown and task-config checks.

The default test suite skips recorded-speech wall-clock replay; run e2e-realtime when changing waveform pacing, caption rendering or replay behavior. Both tasks stay serial because the session fixtures mutate process-wide environment and globals. Real-time acceptance preserves its full original assertions and timing; no accelerated clock is presented as real-time evidence.

For constrained full-Voice testing, use mise run //:test -- extensions/voice --maxWorkers=1 --no-file-parallelism with the environment limits above. Root //:test or //:check supplies complete repository verification at integration boundaries; unrelated CI/browser suites are not implied.

The Node fake lives at __e2e__/fake-native.mjs and must be explicitly selected through voice.command or PI_VOICE_COMMAND. Its service mode is fake; normal startup never substitutes it for a missing executable. It performs logical text/control/event simulation, not microphone capture, synthesis, AEC, model inference or real consumed audio. Optional wire traces require an explicit test-owned OS-temp path and contain transcript/control data. Real-child adapter regressions also use a small scripted peer in __serial__/fixtures/speech-peer.mjs to test precise batching, delayed replies, close-time accounting and fault ordering. That adversarial peer is not another production backend. Delayed-response races use observed requests and test-owned release gates rather than arbitrary sleeps. Mock-only completion signals have bounded deadlines; classifier and recovery timing remain covered with controlled clocks. Consent tests verify both rendered license information and absence of unapproved installation/device requests, including clean child shutdown.

Preserve assertions for provisional versus final input, local/wire identity separation, consumed-prefix history, cue arbitration, mute, cancellation, recovery, setup consent and process ownership. Public Pi-session tests exercise registered commands with an isolated faux conversational provider. Widget/component rendering is not observation of a live terminal. Never claim end-to-end acoustic acceptance from fake captions, waveform bins or playback counters.

Speech fixtures

Fixture provenance records licensed speech/transcript sources and the boundaries of derived evidence. public-speech.json supplies stable clip identities, transcripts, verified checksums and durations; audio binaries stay in OS temp. voice-journeys.json supplies logical long-caption and conversational scenarios. recorded-speech.json supplies a real-time UI replay using measured speech envelopes and synthetic transcript pacing. This replay exercises waveform/provisional/final rendering without a microphone, model, audio decoder or network in the fake. Its generated playback certification is still simulated, not native speech evidence. The ui-preview scenario holds its draft until an explicit test-owned release file exists; preview-race remains the fast protocol-ordering scenario.

Native virtual audio

The native project's TEST_INTERFACE.md proposes the first-class fixture runner; Voice no longer owns a PCM socket/tester or native build tasks. Use that service's explicit fixture launch arguments and manifest after its implementation exists. Prepare approved assets separately; selecting fixture mode does not bypass setup consent. Malformed/unavailable fixtures must fail closed, never enumerate/open physical devices or reconnect implicitly. A native fixture run uses the ordinary production RPC methods and processing path with capture/render endpoints replaced.

Generate or select input once; replay at real time with original pauses/interruptions. Correlate authoritative finals, conversation history and certified playback with indexed recorded PCM. Independently listen to or transcribe output; captions, energy, cues and logical counters alone do not prove correct speech. Use response-consumption-triggered interruption rather than guessed wall-clock delays. Sub-two-second pauses must not dispatch prematurely; two seconds is Voice's endpoint floor, not total response latency. Timing comes from frame positions and the declared sample rate, not notification-arrival bursts. Missing frames, backpressure, disconnects or incomplete recordings invalidate an ordinary run. Injected-fault scenarios pass only by proving their declared fail-closed outcome. Require successful native report/completion plus independent behavioral evidence; recording completion alone is not task success.

Terminal and physical acceptance

Inspect an isolated real Pi screen through terminal-harness. Confirm ready/loading/muted/unavailable cues, provisional/final captions, history, shortcuts and unchanged editor draft. Verify barge-in/hangup stop Voice output without cancelling unrelated Pi work. Keep declared theme/width/container evidence when validating changed visual components.

Native virtual PCM cannot establish physical audibility, real microphone behavior, room AEC or hardware barge-in. Authorize and run those separately on each claimed platform/device. Fake tests establish neither native model quality nor runtime RAM, latency, scheduling or packaging compatibility.

Reporting

Report pass, issues found or blocked, with scenario, service mode/profile, assertions, timings and evidence paths. Separate product defects, harness failures and untested behavior. Distinguish deterministic RPC checks, public command/component integration, observed terminal UX, native fixture calls and physical hardware acceptance. Preserve failure evidence and clean up only test-owned resources.

# Voice verification

## Safety and scope

Preserve active human calls; stopping, reloading or replacing them requires approval.
Use an isolated Pi session, harmless requests and approved model/cache/history locations.
Keep temporary fixtures, traces, relocated artifacts and recordings in OS temp.
Obtain approval before native builds, model downloads/loads, hardware access, diagnostic writes outside approved roots or user configuration changes.
Run checks serially with at least 320 MiB available RAM and `NODE_OPTIONS=--max-old-space-size=160`, `GOMEMLIMIT=96MiB`, `GOMAXPROCS=1`, `RAYON_NUM_THREADS=1`.
Budget native model RAM separately before loading any real speech stack.
Clear obsolete native opt-ins; removed `PI_VOICE_TEST_AUDIO_ENDPOINT` and addon/model environment variables do not activate a Voice test mode.
Declare expected recognition, reply meaning, timing bounds and permitted normalization before testing; never relax them after a failure.

## Fast client checks

Use Mise tasks; no native build, model, hardware or download is needed for deterministic client checks.

- `//extensions/voice:test`: conversation, controller, history, command/UI and RPC regressions.
- `//extensions/voice:e2e`: fast fake-backed public command/RPC scenarios; excludes real-time replay.
- `//extensions/voice:e2e-realtime`: retained real-time recorded-speech UI acceptance, selected explicitly.
- `//extensions/voice:check`: typecheck, static package check and bundled Voice build.
- `//extensions/voice:verify-package`: build and load the relocated bundled client outside the checkout.
- `//extensions/voice:lint`: formatting, Markdown and task-config checks.

The default test suite skips recorded-speech wall-clock replay; run `e2e-realtime` when changing waveform pacing, caption rendering or replay behavior.
Both tasks stay serial because the session fixtures mutate process-wide environment and globals.
Real-time acceptance preserves its full original assertions and timing; no accelerated clock is presented as real-time evidence.

For constrained full-Voice testing, use `mise run //:test -- extensions/voice --maxWorkers=1 --no-file-parallelism` with the environment limits above.
Root `//:test` or `//:check` supplies complete repository verification at integration boundaries; unrelated CI/browser suites are not implied.

The Node fake lives at `__e2e__/fake-native.mjs` and must be explicitly selected through `voice.command` or `PI_VOICE_COMMAND`.
Its service mode is `fake`; normal startup never substitutes it for a missing executable.
It performs logical text/control/event simulation, not microphone capture, synthesis, AEC, model inference or real consumed audio.
Optional wire traces require an explicit test-owned OS-temp path and contain transcript/control data.
Real-child adapter regressions also use a small scripted peer in `__serial__/fixtures/speech-peer.mjs` to test precise batching, delayed replies, close-time accounting and fault ordering.
That adversarial peer is not another production backend.
Delayed-response races use observed requests and test-owned release gates rather than arbitrary sleeps.
Mock-only completion signals have bounded deadlines; classifier and recovery timing remain covered with controlled clocks.
Consent tests verify both rendered license information and absence of unapproved installation/device requests, including clean child shutdown.

Preserve assertions for provisional versus final input, local/wire identity separation, consumed-prefix history, cue arbitration, mute, cancellation, recovery, setup consent and process ownership.
Public Pi-session tests exercise registered commands with an isolated faux conversational provider.
Widget/component rendering is not observation of a live terminal.
Never claim end-to-end acoustic acceptance from fake captions, waveform bins or playback counters.

## Speech fixtures

[Fixture provenance](__e2e__/fixtures/ATTRIBUTION.md) records licensed speech/transcript sources and the boundaries of derived evidence.
`public-speech.json` supplies stable clip identities, transcripts, verified checksums and durations; audio binaries stay in OS temp.
`voice-journeys.json` supplies logical long-caption and conversational scenarios.
`recorded-speech.json` supplies a real-time UI replay using measured speech envelopes and synthetic transcript pacing.
This replay exercises waveform/provisional/final rendering without a microphone, model, audio decoder or network in the fake.
Its generated playback certification is still simulated, not native speech evidence.
The `ui-preview` scenario holds its draft until an explicit test-owned release file exists; `preview-race` remains the fast protocol-ordering scenario.

## Native virtual audio

The native project's `TEST_INTERFACE.md` proposes the first-class fixture runner; Voice no longer owns a PCM socket/tester or native build tasks.
Use that service's explicit fixture launch arguments and manifest after its implementation exists.
Prepare approved assets separately; selecting fixture mode does not bypass setup consent.
Malformed/unavailable fixtures must fail closed, never enumerate/open physical devices or reconnect implicitly.
A native fixture run uses the ordinary production RPC methods and processing path with capture/render endpoints replaced.

Generate or select input once; replay at real time with original pauses/interruptions.
Correlate authoritative finals, conversation history and certified playback with indexed recorded PCM.
Independently listen to or transcribe output; captions, energy, cues and logical counters alone do not prove correct speech.
Use response-consumption-triggered interruption rather than guessed wall-clock delays.
Sub-two-second pauses must not dispatch prematurely; two seconds is Voice's endpoint floor, not total response latency.
Timing comes from frame positions and the declared sample rate, not notification-arrival bursts.
Missing frames, backpressure, disconnects or incomplete recordings invalidate an ordinary run.
Injected-fault scenarios pass only by proving their declared fail-closed outcome.
Require successful native report/completion plus independent behavioral evidence; recording completion alone is not task success.

## Terminal and physical acceptance

Inspect an isolated real Pi screen through `terminal-harness`.
Confirm ready/loading/muted/unavailable cues, provisional/final captions, history, shortcuts and unchanged editor draft.
Verify barge-in/hangup stop Voice output without cancelling unrelated Pi work.
Keep declared theme/width/container evidence when validating changed visual components.

Native virtual PCM cannot establish physical audibility, real microphone behavior, room AEC or hardware barge-in.
Authorize and run those separately on each claimed platform/device.
Fake tests establish neither native model quality nor runtime RAM, latency, scheduling or packaging compatibility.

## Reporting

Report `pass`, `issues found` or `blocked`, with scenario, service mode/profile, assertions, timings and evidence paths.
Separate product defects, harness failures and untested behavior.
Distinguish deterministic RPC checks, public command/component integration, observed terminal UX, native fixture calls and physical hardware acceptance.
Preserve failure evidence and clean up only test-owned resources.