repositories / dotfiles

dotfiles

bugabingas dorkfiles

owned by bugabinga

pi: add mechanical skill validation and controlled behavior checks

c87afe91964b1a0909f05798ecefb189b0bf8460

Oliver Jan Krylow <oliver@bugabinga.net> ·

unsigned

Browse filesCompare with HEAD

Changed files (6)

pi/agent/skills/writing-skills/SKILL.md

Modified
6 6 # Writing Skills
8 -Skills are progressive disclosure: only `name` and `description` are always loaded; `SKILL.md` loads when the task matches.
8 +Skills are progressive disclosure: descriptions select behavior;
9 +bodies load on demand.
10 +Keep durable principles here, not copied API documentation.
10 -## Source of truth
12 +## Preflight
12 -Do not preserve volatile API details in a skill.
13 -For harness-specific behavior, first check the current source of truth: official Agent Skills spec, installed harness docs, `pi --help`/subcommand help, package docs, or local examples.
14 -Put stable principles in the skill; point to the source of truth for details that change.
14 +Before writing, identify:
16 -## Create or refactor
16 +- Outcome, intended trigger, authority, non-goals, and observable success.
17 +- Existing skills/prompts/extensions to reuse;
18 + inspect related policies for contradictions.
19 +- Asset: prompt for a short expansion;
20 + skill for conditional guidance plus references/helpers;
21 + extension only for runtime integration.
22 +- Invocation: automatic matching or manual only.
23 + Check installed Pi docs for `disable-model-invocation` and `/skill:name`;
24 + do not add a redundant dispatcher prompt.
25 +- Placement: global for personal cross-project behavior;
26 + project for repository-specific work;
27 + package for behavior owned by that package.
28 +- Sources: installed docs/help first, then official version-appropriate docs.
29 + Separate durable facts from facts the agent must look up each time.
30 +- Research scratch: an isolated OS temp directory outside repositories.
31 + Persist research elsewhere only when explicitly requested.
32 +- Validation scenario and expected behavior before authoring.
33 + Review requests produce findings, not edits;
34 + obtain approval before implementation.
18 -1. Choose one clear trigger. Split skills when triggers differ.
19 -2. Make `description` specific and assertive. Describe when to load the skill, not the whole workflow.
20 -3. Keep `SKILL.md` short: ordered steps, invariants, and source-of-truth lookup rules.
21 -4. Move bulk examples/reference material into `references/` and load it only when needed.
22 -5. Put deterministic helper logic in `scripts/`; default to cross-platform Node unless the domain requires another runtime.
23 -6. Use relative paths from the skill directory for references, scripts, and assets.
24 -7. Prune duplication, stale recipes, vague completion criteria, and no-op advice.
25 -8. Test activation with a realistic harmless prompt; the model should load the skill and follow it without unintended writes.
36 +## RED → GREEN → REFACTOR
27 -## Shape
38 +1. **RED:** run the harmless scenario without the target skill before creating it.
39 + For edits, also capture current behavior when needed to reproduce the defect.
40 + Record the actual gap; if baseline passes, do not invent failure
41 + or add redundant guidance.
42 +2. **GREEN:** write the minimum skill that closes the observed gap.
43 + Keep one clear trigger, ordered steps, boundaries,
44 + and source lookup rules in `SKILL.md`.
45 + Move optional details into `references/`
46 + and deterministic work into `scripts/`.
47 +3. **REFACTOR:** rerun the same scenario with the skill,
48 + using the same model, settings, and tools.
49 + Inspect skill loading and observable behavior,
50 + not merely the agent's claim of success.
51 + Test a nearby non-trigger and relevant permission/error boundaries.
52 + Prune duplication and rerun affected checks.
29 -```text
30 -my-skill/
31 -├── SKILL.md
32 -├── references/
33 -├── scripts/
34 -└── assets/
35 -```
54 +Use fresh non-interactive Pi runs with `--no-session`
55 +and an explicit tool allowlist.
56 +Disable unrelated extensions, prompts, context files, and skills
57 +for the controlled comparison; enable only the target with `--skill` for GREEN.
58 +Read-only tools are the default;
59 +mutation tests may touch isolated temporary fixtures only.
60 +These flags restrict the test setup, not an OS sandbox;
61 +inspect target instructions and helpers before running them.
62 +Do not launch a visible terminal solely for a headless skill test;
63 +use `terminal-harness` when interaction or terminal UI is under test.
64 +Keep scenario, acceptance criteria, command, model/settings, outputs,
65 +and verdict in temporary evidence.
66 +Report blocked checks and residual gaps explicitly;
67 +never manufacture RED or substitute a static pass for behavioral evidence.
37 -Frontmatter:
69 +## Authoring rules
39 -```yaml
40 ----
41 -name: my-skill
42 -description: "Use when ..."
43 ----
44 -```
71 +- `name`: 1-64 lowercase letters/digits, single internal hyphens;
72 + match the directory for portability.
73 +- `description`: non-empty, at most 1024 characters;
74 + describe when to load, not the entire workflow.
75 +- Use valid YAML quoting or block scalars for punctuation;
76 + hidden/manual skills still need a description.
77 +- Resolve references/scripts/assets relative to the skill directory,
78 + not the caller's cwd.
79 +- Default helpers to cross-platform Node `.mjs`;
80 + use another runtime only for a domain requirement.
81 +- Scripted skills include `package.json`, `npm test` invoking `node --test`,
82 + and isolated `.test.mjs` fixtures.
83 +- Write failing helper regression tests first;
84 + do not execute unreviewed target helpers during an audit.
85 +- Keep runtime secrets, downloaded research,
86 + and generated evidence out of the skill package.
46 -Rules:
88 +## Mechanical checks
48 -- `name`: lowercase letters, numbers, hyphens; 1-64 chars; no leading/trailing/consecutive hyphens.
49 -- `description`: required, <=1024 chars, quoted when it contains punctuation like `:`, `{}`, `[]`, or `#`.
50 -- Directory name should match skill name for portability even when a harness is lenient.
51 -- Skills without `description` do not load.
90 +From this skill directory, install locked parser dependencies once
91 +with `npm ci --ignore-scripts`.
92 +Run `node scripts/check.mjs <skill-dir>`;
93 +see [checker scope](references/checker.md).
94 +It reads only: YAML/name/description, local Markdown links,
95 +and package/test declarations.
96 +It never runs the target's scripts or claims semantic correctness
97 +or test coverage.
53 -## Validation
55 -- Parse frontmatter.
56 -- Check name/description rules.
57 -- Check local relative links exist.
58 -- Run helper-script tests.
59 -- Run formatter/lint for Markdown and package files.
60 -- Run a harmless real-world `pi --print` prompt after refactoring a skill.
99 +Run the target's reviewed tests and repository formatter/linter separately.
100 +Finish with scope, actual RED/GREEN evidence, checks run, and remaining gaps.

pi/agent/skills/writing-skills/package-lock.json

Added
new file mode 100644
1 +{
2 + "name": "writing-skills",
3 + "lockfileVersion": 3,
4 + "requires": true,
5 + "packages": {
6 + "": {
7 + "name": "writing-skills",
8 + "dependencies": {
9 + "marked": "18.0.5",
10 + "yaml": "2.9.0"
11 + },
12 + "engines": {
13 + "node": ">=20"
14 + }
15 + },
16 + "node_modules/marked": {
17 + "version": "18.0.5",
18 + "resolved": "https://registry.npmjs.org/marked/-/marked-18.0.5.tgz",
19 + "integrity": "sha512-S6GcvALHg6K4ohtu4E7x0a1AqhAjp6cV8KhLSyN9qVapnzJkusVBxZRcIU9AeYsbe6P1hKDusSbEOzGyyuce6w==",
20 + "license": "MIT",
21 + "bin": {
22 + "marked": "bin/marked.js"
23 + },
24 + "engines": {
25 + "node": ">= 20"
26 + }
27 + },
28 + "node_modules/yaml": {
29 + "version": "2.9.0",
30 + "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.0.tgz",
31 + "integrity": "sha512-2AvhNX3mb8zd6Zy7INTtSpl1F15HW6Wnqj0srWlkKLcpYl/gMIMJiyuGq2KeI2YFxUPjdlB+3Lc10seMLtL4cA==",
32 + "license": "ISC",
33 + "bin": {
34 + "yaml": "bin.mjs"
35 + },
36 + "engines": {
37 + "node": ">= 14.6"
38 + },
39 + "funding": {
40 + "url": "https://github.com/sponsors/eemeli"
41 + }
42 + }
43 + }

pi/agent/skills/writing-skills/package.json

Added
new file mode 100644
1 +{
2 + "name": "writing-skills",
3 + "private": true,
4 + "type": "module",
5 + "engines": {
6 + "node": ">=20"
7 + },
8 + "scripts": {
9 + "test": "node --test scripts/check.test.mjs"
10 + },
11 + "dependencies": {
12 + "marked": "18.0.5",
13 + "yaml": "2.9.0"
14 + }

pi/agent/skills/writing-skills/references/checker.md

Added
new file mode 100644
1 +# Checker scope
3 +Run from the writing-skills directory:
5 +```sh
6 +npm ci --ignore-scripts
7 +node scripts/check.mjs /path/to/target-skill
8 +npm test
9 +```
11 +`npm ci` installs this checker's locked `yaml` and `marked` parsers,
12 +not target dependencies.
13 +The checker accepts one skill directory containing `SKILL.md`,
14 +not a discovery root or loose Markdown skill.
15 +It works from any cwd when the script and target paths resolve correctly.
17 +## Checks
19 +- Required YAML fences and a valid mapping, including duplicate-key rejection.
20 +- String name and description, length limits, portable name syntax,
21 + and directory/name agreement.
22 +- Boolean `disable-model-invocation` when supplied; unknown fields are allowed.
23 +- Inline, image, and reference-style local links
24 + in regular `.md` files throughout the target.
25 + Resolve relative to each source document;
26 + ignore code examples, URLs, absolute links, and anchors.
27 + Check file existence, not heading anchors, raw HTML links,
28 + or remote availability.
29 +- Scripted skills require `package.json`, a direct `node --test` test command,
30 + and at least one `.test.mjs` file.
31 + Recognize `.mjs`, `.cjs`, `.js`, `.ts`, `.sh`, `.bash`, `.nu`, `.py`, `.ps1`,
32 + `.cmd`, and `.bat` helpers anywhere in the target.
33 + Do not infer executable behavior for extensionless files.
34 +- Skip `node_modules`, `.git`, and directory/file symlinks
35 + during recursive enumeration.
36 + Read the explicitly selected `SKILL.md`;
37 + linked targets are checked for existence, not recursively audited.
39 +Directory/name matching and script packaging are authoring policy,
40 +not claims about what Pi accepts.
41 +The checker does not evaluate helper code, shell commands, coverage,
42 +trigger quality, policy consistency, or correctness of prose.
43 +A test declaration plus test file is not proof
44 +that the command selects or passes those tests.
45 +Review and run the target's actual tests separately.
47 +## Output
49 +JSON contains `skill`, `ok`, and `issues`
50 +with source `path`, stable `code`, and `message`.
51 +Exit codes: `0` means checks passed; `1` means findings;
52 +`2` means usage or filesystem failure.
53 +A failure is not a clean audit.
54 +Do not copy raw findings into public artifacts
55 +without checking for sensitive paths or URLs.

pi/agent/skills/writing-skills/scripts/check.mjs

Added
new file mode 100644
1 +#!/usr/bin/env node
2 +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs";
3 +import {
4 + basename,
5 + dirname,
6 + isAbsolute,
7 + join,
8 + relative,
9 + resolve,
10 +} from "node:path";
11 +import { fileURLToPath } from "node:url";
12 +import { marked } from "marked";
13 +import { parse } from "yaml";
15 +const frontmatterFence = /^\uFEFF?---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/;
16 +const namePattern = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
17 +const scriptPattern = /\.(?:mjs|cjs|js|ts|sh|bash|nu|py|ps1|cmd|bat)$/i;
19 +function filesUnder(dir) {
20 + return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => {
21 + if (entry.name === "node_modules" || entry.name === ".git") return [];
22 + const path = join(dir, entry.name);
23 + if (entry.isDirectory()) return filesUnder(path);
24 + return entry.isFile() ? [path] : [];
25 + });
28 +export function checkSkill(directory) {
29 + const root = resolve(directory);
30 + if (!statSync(root).isDirectory())
31 + throw new Error("Expected a skill directory");
32 + const skill = join(root, "SKILL.md");
33 + const text = readFileSync(skill, "utf8");
34 + const issues = [];
35 + const add = (path, code, message) =>
36 + issues.push({
37 + path: relative(root, path).split("\\").join("/"),
38 + code,
39 + message,
40 + });
41 + const fence = text.match(frontmatterFence);
42 + if (!fence) {
43 + add(
44 + skill,
45 + "frontmatter.fence",
46 + "Expected opening and closing YAML frontmatter fences",
47 + );
48 + } else {
49 + let data;
50 + try {
51 + data = parse(fence[1]);
52 + } catch {
53 + add(
54 + skill,
55 + "frontmatter.yaml",
56 + "Invalid YAML frontmatter, keys, or aliases",
57 + );
58 + }
59 + if (data !== undefined) {
60 + if (!data || typeof data !== "object" || Array.isArray(data)) {
61 + add(skill, "frontmatter.mapping", "Frontmatter must be a YAML mapping");
62 + } else {
63 + const { name, description } = data;
64 + if (
65 + typeof name !== "string" ||
66 + name.length > 64 ||
67 + !namePattern.test(name)
68 + ) {
69 + add(
70 + skill,
71 + "name.invalid",
72 + "Name must be 1-64 lowercase letters/digits with single internal hyphens",
73 + );
74 + } else if (name !== basename(root)) {
75 + add(
76 + skill,
77 + "name.directory",
78 + "Name must match directory for Agent Skills portability",
79 + );
80 + }
81 + if (
82 + typeof description !== "string" ||
83 + !description.trim() ||
84 + description.length > 1024
85 + ) {
86 + add(
87 + skill,
88 + "description.invalid",
89 + "Description must be a non-empty string of at most 1024 characters",
90 + );
91 + }
92 + const invocation = data["disable-model-invocation"];
93 + if (invocation !== undefined && typeof invocation !== "boolean") {
94 + add(
95 + skill,
96 + "invocation.invalid",
97 + "disable-model-invocation must be a boolean when present",
98 + );
99 + }
104 + const files = filesUnder(root).sort();
105 + for (const file of files.filter((path) => path.endsWith(".md"))) {
106 + const body = readFileSync(file, "utf8").replace(frontmatterFence, "");
107 + marked.walkTokens(marked.lexer(body), (token) => {
108 + if (token.type !== "link" && token.type !== "image") return;
109 + const href = token.href;
110 + if (
111 + !href ||
112 + /^[a-z][a-z\d+.-]*:|^[#/?\\]/i.test(href) ||
113 + isAbsolute(href)
115 + return;
116 + let target;
117 + try {
118 + target = decodeURIComponent(href.split(/[?#]/, 1)[0]);
119 + } catch {
120 + add(file, "link.encoding", "Local link has invalid percent encoding");
121 + return;
123 + if (!existsSync(resolve(dirname(file), target))) {
124 + add(file, "link.missing", `Missing local link target: ${href}`);
126 + });
129 + const scripted = files.some((file) => scriptPattern.test(file));
130 + const packageFile = join(root, "package.json");
131 + if (!existsSync(packageFile)) {
132 + if (scripted)
133 + add(
134 + packageFile,
135 + "package.missing",
136 + "Scripted skills require package.json",
137 + );
138 + } else {
139 + let manifest;
140 + try {
141 + manifest = JSON.parse(readFileSync(packageFile, "utf8"));
142 + } catch {
143 + manifest = null;
145 + if (!manifest || typeof manifest !== "object" || Array.isArray(manifest)) {
146 + add(
147 + packageFile,
148 + "package.json",
149 + "Expected a JSON object in package.json",
150 + );
151 + } else if (
152 + scripted &&
153 + (typeof manifest.scripts?.test !== "string" ||
154 + !/^node\s+--test(?:\s|$)/.test(manifest.scripts.test))
155 + ) {
156 + add(
157 + packageFile,
158 + "tests.command",
159 + "scripts.test must invoke node --test directly",
160 + );
163 + if (scripted && !files.some((file) => file.endsWith(".test.mjs"))) {
164 + add(
165 + packageFile,
166 + "tests.missing",
167 + "Scripted skills require at least one .test.mjs file",
168 + );
170 + return { skill: root, ok: issues.length === 0, issues };
173 +if (
174 + process.argv[1] &&
175 + resolve(process.argv[1]) === fileURLToPath(import.meta.url)
176 +) {
177 + try {
178 + if (process.argv.length !== 3)
179 + throw new Error("Usage: node scripts/check.mjs <skill-dir>");
180 + const result = checkSkill(process.argv[2]);
181 + console.log(JSON.stringify(result, null, 2));
182 + process.exitCode = result.ok ? 0 : 1;
183 + } catch (error) {
184 + console.log(JSON.stringify({ error: error.message }));
185 + process.exitCode = 2;

pi/agent/skills/writing-skills/scripts/check.test.mjs

Added
new file mode 100644
1 +import assert from "node:assert/strict";
2 +import { spawnSync } from "node:child_process";
3 +import {
4 + mkdirSync,
5 + mkdtempSync,
6 + readFileSync,
7 + rmSync,
8 + symlinkSync,
9 + writeFileSync,
10 +} from "node:fs";
11 +import { tmpdir } from "node:os";
12 +import { dirname, join } from "node:path";
13 +import { test } from "node:test";
14 +import { fileURLToPath } from "node:url";
15 +import { checkSkill } from "./check.mjs";
17 +const cli = fileURLToPath(new URL("./check.mjs", import.meta.url));
18 +const header = '---\nname: example\ndescription: "Use when testing."\n---\n';
20 +function fixture(t, files = {}) {
21 + const temp = mkdtempSync(join(tmpdir(), "writing skills-"));
22 + t.after(() => rmSync(temp, { recursive: true, force: true }));
23 + const root = join(temp, "example");
24 + for (const [path, content] of Object.entries({
25 + "SKILL.md": header,
26 + ...files,
27 + })) {
28 + const dest = join(root, path);
29 + mkdirSync(dirname(dest), { recursive: true });
30 + writeFileSync(dest, content);
31 + }
32 + return root;
35 +function codes(root) {
36 + return checkSkill(root).issues.map((issue) => issue.code);
39 +test("valid prose-only skill needs neither package nor tests", (t) => {
40 + const result = checkSkill(fixture(t));
41 + assert.equal(result.ok, true);
42 + assert.deepEqual(result.issues, []);
43 +});
45 +for (const [label, content, expected] of [
46 + ["missing fences", "name: example", "frontmatter.fence"],
47 + ["unclosed fence", "---\nname: example", "frontmatter.fence"],
48 + ["invalid YAML", "---\nname: [\n---\n", "frontmatter.yaml"],
49 + [
50 + "duplicate YAML key",
51 + header.replace("name: example", "name: example\nname: other"),
52 + "frontmatter.yaml",
53 + ],
54 + ["non-mapping YAML", "---\n- example\n---\n", "frontmatter.mapping"],
55 + ["missing name", header.replace("name: example\n", ""), "name.invalid"],
56 + ["non-string name", header.replace("example", "123"), "name.invalid"],
57 + [
58 + "consecutive hyphens",
59 + header.replace("example", "ex--ample"),
60 + "name.invalid",
61 + ],
62 + ["uppercase name", header.replace("example", "Example"), "name.invalid"],
63 + ["leading hyphen", header.replace("example", "-example"), "name.invalid"],
64 + ["trailing hyphen", header.replace("example", "example-"), "name.invalid"],
65 + ["long name", header.replace("example", "a".repeat(65)), "name.invalid"],
66 + ["directory mismatch", header.replace("example", "other"), "name.directory"],
67 + ["missing description", "---\nname: example\n---\n", "description.invalid"],
68 + [
69 + "empty description",
70 + header.replace('"Use when testing."', '" "'),
71 + "description.invalid",
72 + ],
73 + [
74 + "non-string description",
75 + header.replace('"Use when testing."', "false"),
76 + "description.invalid",
77 + ],
78 + [
79 + "long description",
80 + header.replace("Use when testing.", "x".repeat(1025)),
81 + "description.invalid",
82 + ],
83 + [
84 + "non-boolean invocation flag",
85 + header.replace(
86 + "name: example",
87 + 'name: example\ndisable-model-invocation: "false"',
88 + ),
89 + "invocation.invalid",
90 + ],
91 +]) {
92 + test(label, (t) =>
93 + assert.ok(codes(fixture(t, { "SKILL.md": content })).includes(expected)),
94 + );
97 +test("YAML folded descriptions, comments, CRLF and unknown metadata are valid", (t) => {
98 + const content =
99 + "---\nname: example # comment\ndescription: >-\n Use when testing:\n quoted punctuation is not required here.\ndisable-model-invocation: true\nmetadata:\n topic: checks\n---\n";
100 + assert.deepEqual(
101 + codes(
102 + fixture(t, { "SKILL.md": `\uFEFF${content.replaceAll("\n", "\r\n")}` }),
103 + ),
104 + [],
105 + );
106 +});
108 +test("YAML scalar aliases resolve", (t) => {
109 + const root = fixture(t, {
110 + "SKILL.md": header.replace(
111 + 'description: "Use when testing."',
112 + 'label: &label "Use when testing."\ndescription: *label',
113 + ),
114 + });
115 + assert.deepEqual(codes(root), []);
116 +});
118 +test("unresolved YAML aliases fail even in optional metadata", (t) => {
119 + const root = fixture(t, {
120 + "SKILL.md": header.replace(
121 + "name: example",
122 + "name: example\nmetadata: *missing",
123 + ),
124 + });
125 + assert.ok(codes(root).includes("frontmatter.yaml"));
126 +});
128 +test("local inline, image, reference and nested Markdown links resolve per document", (t) => {
129 + const root = fixture(t, {
130 + "SKILL.md": `${header}\n[ref](references/guide.md#part)\n![image](assets/pic.png)\n[space][s]\n\n[s]: <references/a b.md>\n`,
131 + "references/guide.md":
132 + "[back](../SKILL.md)\n[encoded](a%20b.md?view=1#part)\n[parenthesized](a(b).md)",
133 + "references/a b.md": "OK",
134 + "references/a(b).md": "OK",
135 + "assets/pic.png": "fixture",
136 + });
137 + assert.deepEqual(codes(root), []);
138 +});
140 +test("missing local links report their source file", (t) => {
141 + const root = fixture(t, {
142 + "references/guide.md": "[missing](gone.md)\n![missing](gone.png)",
143 + });
144 + const result = checkSkill(root);
145 + assert.equal(result.ok, false);
146 + assert.deepEqual(
147 + result.issues.map(({ path, code }) => [path, code]),
149 + ["references/guide.md", "link.missing"],
150 + ["references/guide.md", "link.missing"],
151 + ],
152 + );
153 +});
155 +test("external URLs, anchors, absolute links and code examples are not local-link checks", (t) => {
156 + const root = fixture(t, {
157 + "SKILL.md": `${header}
158 +[web](https://example.test/missing)
159 +[mail](mailto:test@example.test)
160 +[anchor](#missing)
161 +[absolute](/missing)
162 +[windows](C:/missing)
163 +[network](//example.test/missing)
164 +\`[code](missing.md)\`
166 +\`\`\`md
167 +[example](missing.md)
168 +\`\`\`
170 + });
171 + assert.deepEqual(codes(root), []);
172 +});
174 +test("bad URL encoding is an actionable issue", (t) => {
175 + assert.ok(
176 + codes(fixture(t, { "SKILL.md": `${header}[bad](missing%ZZ.md)` })).includes(
177 + "link.encoding",
178 + ),
179 + );
180 +});
182 +test("dependencies and VCS internals are not traversed", (t) => {
183 + assert.deepEqual(
184 + codes(
185 + fixture(t, {
186 + "node_modules/pkg/SKILL.md": "[bad](gone)",
187 + ".git/internal.md": "[bad](gone)",
188 + }),
189 + ),
190 + [],
191 + );
192 +});
194 +test("directory symlinks are not traversed", (t) => {
195 + const root = fixture(t);
196 + const external = join(dirname(root), "outside");
197 + mkdirSync(external);
198 + writeFileSync(join(external, "bad.md"), "[bad](gone)");
199 + symlinkSync(
200 + external,
201 + join(root, "references"),
202 + process.platform === "win32" ? "junction" : "dir",
203 + );
204 + assert.deepEqual(codes(root), []);
205 +});
207 +const helper = {
208 + "scripts/helper.mjs": 'throw new Error("must never execute");',
210 +const testFile = {
211 + "scripts/helper.test.mjs": 'throw new Error("must never execute");',
213 +const manifest = (command) => JSON.stringify({ scripts: { test: command } });
215 +for (const [label, files, expected] of [
216 + ["missing package and tests", helper, ["package.missing", "tests.missing"]],
218 + "bad JSON",
219 + { ...helper, ...testFile, "package.json": "{" },
220 + ["package.json"],
221 + ],
223 + "non-object manifest",
224 + { ...helper, ...testFile, "package.json": "null" },
225 + ["package.json"],
226 + ],
228 + "missing test command",
229 + { ...helper, ...testFile, "package.json": "{}" },
230 + ["tests.command"],
231 + ],
233 + "non-string test command",
234 + { ...helper, ...testFile, "package.json": manifest(42) },
235 + ["tests.command"],
236 + ],
238 + "fake test command",
239 + { ...helper, ...testFile, "package.json": manifest("echo node --test") },
240 + ["tests.command"],
241 + ],
243 + "missing tests",
244 + { ...helper, "package.json": manifest("node --test") },
245 + ["tests.missing"],
246 + ],
248 + "valid scripted skill",
250 + ...helper,
251 + ...testFile,
252 + "package.json": manifest("node --test scripts/*.test.mjs"),
253 + },
254 + [],
255 + ],
257 + "nested tests",
259 + ...helper,
260 + "test/nested/helper.test.mjs": "",
261 + "package.json": manifest("node --test"),
262 + },
263 + [],
264 + ],
266 + "domain-specific helper",
268 + "scripts/helper.ps1": "",
269 + ...testFile,
270 + "package.json": manifest("node --test"),
271 + },
272 + [],
273 + ],
274 +]) {
275 + test(label, (t) => assert.deepEqual(codes(fixture(t, files)), expected));
278 +test("CLI succeeds from unrelated cwd and paths with spaces", (t) => {
279 + const root = fixture(t, { "references/path with spaces.md": "OK" });
280 + const result = spawnSync(process.execPath, [cli, root], {
281 + cwd: tmpdir(),
282 + encoding: "utf8",
283 + });
284 + assert.equal(result.status, 0, result.stderr);
285 + assert.equal(JSON.parse(result.stdout).ok, true);
286 +});
288 +test("CLI returns 1 for findings", (t) => {
289 + const result = spawnSync(
290 + process.execPath,
291 + [cli, fixture(t, { "SKILL.md": "bad" })],
292 + { encoding: "utf8" },
293 + );
294 + assert.equal(result.status, 1);
295 + assert.equal(JSON.parse(result.stdout).ok, false);
296 +});
298 +test("CLI returns 2 for usage or inaccessible input", (t) => {
299 + const root = fixture(t);
300 + for (const args of [[], [root, "extra"], [join(root, "missing")]]) {
301 + const result = spawnSync(process.execPath, [cli, ...args], {
302 + encoding: "utf8",
303 + });
304 + assert.equal(result.status, 2, result.stderr);
305 + assert.ok(JSON.parse(result.stdout).error);
307 +});
309 +test("checker audits itself", () => {
310 + const root = fileURLToPath(new URL("..", import.meta.url));
311 + assert.deepEqual(codes(root), []);
312 + assert.ok(readFileSync(join(root, "SKILL.md"), "utf8").includes("check.mjs"));
313 +});