pi-ext bugabingas pi extensions
owned by admin
README Source History Refs Compare Notes Search fix(ultra): separate performance from validation
86c9367d22075cbfd8329f51100ce940b5cbf569
Oliver Jan Krylow <oliver@bugabinga.net > · 27 Aug 2026
SSH signature · not checked
Browse files Compare with HEAD
Changed files (7) 19 19 - Let loaded extensions activate their own Ultra sub-agent tools without extension-specific workflow tool names
20 20 - Let Ultra create, check out, validate, publish, reuse, and load dynamic extensions from a canonical catalog under Pi's agent directory; keep one accepted version per name, isolate editable drafts in the platform cache, reject stale updates, retry deliberate rebases once, and publish atomically
21 21 - Treat manually edited canonical dynamic extensions as unvalidated until they are validated in place, and require revalidation when the validator contract changes
22 - - Validate dynamic extension drafts with real Pi loading, `node:test` capability coverage, structural checks, deterministic network-free tests, and hot-path benchmarks; warn on slow validation without encouraging test removal, enforce load and benchmark budgets, report actionable diagnostics for every independent failure, and retain unpublished drafts when validation is cancelled
22 + - Validate dynamic extension drafts with real Pi loading, `node:test` capability coverage, structural checks, deterministic network-free tests, and hot-path benchmarks; report load and benchmark overruns as advisory warnings, report actionable diagnostics for every independent correctness failure, and retain unpublished drafts when validation is cancelled
23 23 - Give dynamic-extension builders every available tool except recursive Ultra workflow execution while keeping consumer agents unaware of the build system
24 24 - Load one or more dynamic extensions after configured extensions, resolve stable names to recorded source hashes for each run, preserve declared order, require explicit qualified override metadata for collisions, and block consumer phases when extension building fails
25 25 - Let Ultra repair dynamic extensions after consumer failures and rerun the original workflow, reusing unaffected checkpoints without a fixed repair-cycle limit
extensions/ultra/U008.md Modified 164 - `validate_dynamic_extension` runs mandatory gates, produces actionable diagnostics, and publishes only on complete success.
165 - Failure retains the draft and publishes nothing.
164 + `validate_dynamic_extension` runs mandatory correctness gates and advisory performance checks.
165 + Correctness failure retains the draft and publishes nothing.
166 + Performance budget overruns produce warnings and do not block publication because scheduler load is not correctness evidence.
166 167 Success returns stable name, full revision hash, description, and exact canonical source path.
168 169 Diagnostics report all independent failures in one call.
340 341 4. fixed `debug.ts` template
341 342 5. symlink rejection and deterministic whole-artifact hashing
342 343 6. all `node:test` files with capability mapping
343 - 7. required `benchmark.ts` contract and current-platform hot-path budget
344 + 7. required `benchmark.ts` contract and advisory current-platform hot-path budget comparison
344 345 8. real Pi SDK loading through scoped `DefaultResourceLoader`, in-memory `AgentSession`, `bindExtensions({ mode: "print" })`, `session_start`, registration inspection, and `session_shutdown`
345 346 9. active generated-tool verification
346 - 10. marginal extension load timing
347 + 10. advisory marginal extension load timing
347 348 11. attestation generation and canonical publication
349 350 Real loading uses Pi SDK directly and makes no model call.
359 360 Test runtime does not fail merely for exceeding the five-second target, because a hard short target would reward deleting tests.
360 361 Capability mapping remains mandatory.
361 - Only the watchdog, marginal load budget, and declared hot-path budgets are timing failures.
362 + The watchdog remains a deadlock ceiling.
363 + Marginal load and hot-path budget overruns are warnings because benchmark results require controlled conditions.
363 365 Validator-contract version participates in attestation validity.
364 366 After a contract update, search marks old attestations outdated and loading requires revalidation.
extensions/ultra/__tests__/dynamic-extensions.test.ts Modified 547 547 expect (builder.activeTools).
toContain (
"read" );
548 548 expect (builder.activeTools).
toContain (
"edit" );
549 549 expect (builder.activeTools).not.
toContain (
"run_workflow" );
550 - expect (builder.
prompt ).
toContain (
"builder contract v1" );
550 + expect (builder.
prompt ).
toContain (
"builder contract v2" );
551 551 const byName
= new Map (builder.tools.
map ((tool)
=> [tool.
name , tool]));
552 552 const search
= await invoke (
553 553 requiredTool (byName,
"search_dynamic_extensions" ),
574 574 expect (
await readFile (
join (created.draftPath,
"debug.ts" ),
"utf8" )).
toBe (
575 575 DYNAMIC_DEBUG_TEMPLATE,
577 + const manifestPath
= join (created.draftPath,
"manifest.json" );
578 + const manifest
= JSON .
parse (
await readFile (manifestPath,
"utf8" ))
as {
579 + benchmarkBudgets
: Record <
string , { hotPathMs
: number }>;
581 + manifest.benchmarkBudgets[
process .
platform ]
= {
582 + hotPathMs:
Number .EPSILON,
584 + await writeFile (manifestPath,
`${ JSON . stringify (manifest, null, 2)}\n` );
577 585 const validation
= (
await invoke (
578 586 requiredTool (byName,
"validate_dynamic_extension" ),
579 587 { path: created.draftPath },
580 - ))
as Record <
string ,
unknown >;
591 + diagnostics
: Array <{ code
: string ; status
: string }>;
581 593 expect (validation,
JSON .
stringify (validation,
null ,
2 )).
toMatchObject ({
582 594 status:
"validated" ,
583 595 name:
"probe-capability" ,
597 + expect (validation.diagnostics).
toEqual (
598 + expect.
arrayContaining ([
599 + expect.
objectContaining ({
600 + code:
"benchmark_budget" ,
585 605 const resolved
= await coordinator.
resolve ([
"probe-capability" ], {
586 606 phase:
"consume" ,
587 607 agentId:
"consume#0" ,
672 692 await coordinator.
complete ();
673 693 expect (
await stat (resolved[
0 ].sourcePath)).
toBeDefined ();
674 694 await expect (
stat (brokenCheckout.draftPath)).rejects.
toThrow ();
677 697 it (
"serializes concurrent publication and rejects the stale builder without clobbering" ,
async ()
=> {
678 698 const root
= await mkdtemp (
join (
tmpdir (),
"ultra-stale-test-" ));
679 699 roots.
push (
root );
680 700 const paths
= fakePaths (
root );
701 + const workerPath
= join (
root ,
"successful-validator.mjs" );
704 + `import { readFile, writeFile } from "node:fs/promises"; 705 + import { join } from "node:path"; 706 + import { hashArtifacts } from ${ JSON . stringify ( new URL ("../dynamic-artifacts.ts", import . meta .url). href )}; 707 + const draftPath = process.argv[2]; 708 + const manifest = JSON.parse(await readFile(join(draftPath, "manifest.json"), "utf8")); 709 + const hash = hashArtifacts(draftPath); 717 + validatorContractVersion: ${VALIDATOR_CONTRACT_VERSION}, 719 + validatedAt: new Date().toISOString(), 720 + platform: process.platform, 721 + testedPlatforms: [process.platform], 722 + nodeVersion: process.version, 723 + warningTargetMs: 5_000, 728 + await writeFile(process.env.ULTRA_VALIDATOR_RESULT_PATH, JSON.stringify(result), "utf8"); 681 732 const coordinator
= createDynamicExtensionCoordinator ({
682 733 agentDir:
join (
root ,
"agent" ),
683 734 parentSessionId:
"parent" ,
684 735 workflowRunId:
"stale-run" ,
685 736 availableTools: [
"read" ,
"edit" ],
688 740 const initial
= await coordinator.
builder (
"initial#0" );
689 741 const initialTools
= new Map (
742 794 path : rightCheckout.draftPath,
744 796 ]))
as Array <
Record <
string ,
unknown >>;
745 - expect (publications.
map ((value)
=> value.
status ).
sort ()).
toEqual ([
798 + publications.
map ((value)
=> value.
status ).
sort (),
799 + JSON .
stringify (publications,
null ,
2 ),
800 + ).
toEqual ([
"stale" ,
"validated" ]);
749 801 const canonical
= await coordinator.
resolve ([
"stale-probe-capability" ], {
750 802 phase:
"consume" ,
751 803 agentId:
"consume#0" ,
extensions/ultra/dynamic-catalog.ts Modified 686 686 name :
"validate_dynamic_extension" ,
687 687 label :
"Validate dynamic extension" ,
689 - "Run every mandatory validation gate and publish atomically only on complete success." ,
689 + "Run mandatory correctness gates and advisory performance checks, then publish atomically only when correctness succeeds." ,
690 690 parameters : Type.
Object (
691 691 { path: Type.
String ({ minLength:
1 }) },
692 692 { additionalProperties:
false },
extensions/ultra/dynamic-contract.ts Modified 1 - export const VALIDATOR_CONTRACT_VERSION
= 1 ;
1 + export const VALIDATOR_CONTRACT_VERSION
= 2 ;
2 2 export const VALIDATOR_WARNING_TARGET_MS
= 5_000 ;
3 3 export const VALIDATOR_WATCHDOG_MS
= 15_000 ;
4 4 export const MARGINAL_LOAD_BUDGET_MS
= 50 ;
extensions/ultra/dynamic-validator-worker.ts Modified 802 802 diagnostic (state, {
803 803 gate:
"benchmark" ,
804 804 code:
"benchmark_budget" ,
807 807 expected: budget,
808 808 actual: measured,
810 - "Optimize the hot path until it meets the declared current-platform budget." ,
810 + "Reproduce under controlled benchmark conditions, then optimize the hot path or deliberately revise its budget." ,
875 875 diagnostic (state, {
876 876 gate:
"load-timing" ,
877 877 code:
"marginal_load_budget" ,
880 880 limitMs: MARGINAL_LOAD_BUDGET_MS,
881 881 elapsedMs: loadMs,
883 - "Make index.ts thin and defer optional work until first use." ,
883 + "Reproduce under controlled benchmark conditions, then make index.ts thin and defer optional work until first use." ,
885 885 const model
= getModels (
"openai" )[
0 ]
?? getModels (
"anthropic" )[
0 ];
extensions/ultra/prompts/dynamic-extension-builder.md Modified 1 - #
Ultra dynamic-extension builder contract v1 1 + #
Ultra dynamic-extension builder contract v2 3 3 You own exactly one purpose-built Pi extension topic.
4 4 The extension has the orchestrator's full host authority and is not sandboxed.
30 30 - Declare every intentional collision as an exact qualified override such as `tool:read`.
31 31 - Never register or override Ultra's reserved `run_workflow`, `structured_output`, or four dynamic-extension builder tool names.
32 32 - Preserve or tighten existing platform hot-path budgets; never loosen them.
33 + - Treat load and hot-path budget warnings as benchmark evidence to reproduce under controlled conditions, not correctness failures.
33 34 - Keep Markdown prose semantic, one sentence per line.
35 36 Use `big/high` for new extension architecture and `medium/high` for bounded improvements unless configured models or task economics justify another deliberate choice.
36 - Validation runs all independent gates, retains failed or aborted drafts, and publishes only complete success.
37 + Validation runs all independent correctness gates plus advisory performance checks, retains failed or aborted drafts, and publishes only when correctness succeeds.