Use if task creates/runs throwaway code/data/artifacts:
scratch scripts/files, spikes, probes, PoCs
microbenchmarks, migration dry-runs
one-off parsers/converters/generators
API/dependency/env behavior checks
temp fixtures/logs/output dirs to answer 1 question
Do not use for normal impl/review/tests except exploratory phase.
Protocol
1. Hypothesis
Before code/run:
Q: ...
H: if <condition/change/input> → <observable> because <reason>
Falsifier: false if <specific obs>
Control/baseline: ...
Cmd/input/env: ...
Artifacts: <temp paths>
External retention: keep/delete <temp, cache, or OS state> because <value>
Unfalsifiable H ⇒ rewrite.
2. Isolation
Always OS temp: one mktemp -d dir per experiment, prefix pi-exp..
Never create scratch dirs in the repo, ignored or not (.pi/tmp, tmp/, .test-home, probe plugins).
Tool needs repo-relative paths → point it at the temp dir via flags/env; impossible → ask first.
git status --short before+after; any new repo path = defect.
3. Provenance
Record if result matters:
exact cmd(s), inputs, sample data
tool/runtime/dependency versions; lockfile state
env/config/flags/params
random seed(s); nondeterminism notes
commit hash if git
timing context:
hw/load/warmup/repeats
Prefer executable protocol over prose:
cmd/test/mise task/script/fixture.
4. Verification ladder
Use ≥2 independent checks when possible.
Predict output before run.
Positive control:
known-good passes.
Negative control:
known-bad fails.
Edge cases:
empty/min/max/weird.
Assertions/invariants fail loud.
Cross-check via 2nd method/tool/manual sample.
Rerun:
deterministic or bounded variance.
Seed randomness or report variance.
Compare baseline/current vs changed.
Explain mismatch; unresolved ⇒ inconclusive.
Taste/UX outcomes: show the artifact to the user; never invent numeric thresholds.
1 green run = weak evidence.
Log/screenshot without cmd provenance = weak.
Summarize:
learned fact only → response/issue/commit msg.
Retain:
useful temp/cache/OS artifacts outside the repo.
Record location, purpose, and expected lifetime.
Delete:
temp artifacts without future value.
Promotion requirements:
named project location; no local abs paths/debug prints
minimal fixture size
conforms to fmt/lint/test conventions
Cleanup
Before final answer:
Stop processes that are no longer useful.
Delete temp artifacts without future value; preserve useful and safe temp, cache, or OS state.
Report retained state and ask if its footprint or side effects are uncertain.
git status --short if git; any experiment-created repo path is a defect to remove.
Delete rule:
Only delete explicit paths created by this experiment.
Never delete user data, research cache, repo files, broad globs, or git-clean
without confirmation.
Cleanup means leaving the user's repository clean, not erasing useful external
experimental state.
broad cleanup:
rm -rf *, git clean -fdx sans consent
---
name: experimentation
description: "Use when asked to do an experiment."
---
# Experimentation
Temp code = experiment.
Learn → verify → preserve useful evidence → leave the repo clean.
## Trigger
Use if task creates/runs throwaway code/data/artifacts:
- scratch scripts/files, spikes, probes, PoCs
- microbenchmarks, migration dry-runs
- one-off parsers/converters/generators
- API/dependency/env behavior checks
- temp fixtures/logs/output dirs to answer 1 question
Do **not** use for normal impl/review/tests except exploratory phase.
## Protocol
### 1. Hypothesis
Before code/run:
```text
Q: ...
H: if <condition/change/input> → <observable> because <reason>
Falsifier: false if <specific obs>
Control/baseline: ...
Cmd/input/env: ...
Artifacts: <temp paths>
External retention: keep/delete <temp, cache, or OS state> because <value>
```
Unfalsifiable H ⇒ rewrite.
### 2. Isolation
Always OS temp: one `mktemp -d` dir per experiment, prefix `pi-exp.`.
Never create scratch dirs in the repo, ignored or not (`.pi/tmp`, `tmp/`, `.test-home`, probe plugins).
Tool needs repo-relative paths → point it at the temp dir via flags/env; impossible → ask first.
`git status --short` before+after; any new repo path = defect.
### 3. Provenance
Record if result matters:
- exact cmd(s), inputs, sample data
- tool/runtime/dependency versions; lockfile state
- env/config/flags/params
- random seed(s); nondeterminism notes
- commit hash if git
- timing context:
hw/load/warmup/repeats
Prefer executable protocol over prose:
cmd/test/mise task/script/fixture.
### 4. Verification ladder
Use ≥2 independent checks when possible.
1. Predict output before run.
2. Positive control:
known-good passes.
3. Negative control:
known-bad fails.
4. Edge cases:
empty/min/max/weird.
5. Assertions/invariants fail loud.
6. Cross-check via 2nd method/tool/manual sample.
7. Rerun:
deterministic or bounded variance.
8. Seed randomness or report variance.
9. Compare baseline/current vs changed.
10. Explain mismatch; unresolved ⇒ inconclusive.
Taste/UX outcomes: show the artifact to the user; never invent numeric thresholds.
1 green run = weak evidence.
Log/screenshot without cmd provenance = weak.
### 5. Promote | summarize | retain | delete
Classify every artifact:
- **Promote**:
durable test/fixture/doc/bench harness/script/repro.
- **Summarize**:
learned fact only → response/issue/commit msg.
- **Retain**:
useful temp/cache/OS artifacts outside the repo.
Record location, purpose, and expected lifetime.
- **Delete**:
temp artifacts without future value.
Promotion requirements:
- named project location; no local abs paths/debug prints
- minimal fixture size
- conforms to fmt/lint/test conventions
## Cleanup
Before final answer:
1. Stop processes that are no longer useful.
2. Delete temp artifacts without future value; preserve useful and safe temp, cache, or OS state.
3. Report retained state and ask if its footprint or side effects are uncertain.
4. `git status --short` if git; any experiment-created repo path is a defect to remove.
Delete rule:
```text
Only delete explicit paths created by this experiment.
Never delete user data, research cache, repo files, broad globs, or git-clean
without confirmation.
Cleanup means leaving the user's repository clean, not erasing useful external
experimental state.
```
Final report shape:
```text
Result: supported | falsified | inconclusive
Evidence: cmd(s) + checks
Promoted: <paths + why>
Retained: <external paths/state + why>
Cleaned: <paths>
```
## Smells
- any experiment-created path inside the repo
- hidden generated dir no owner
- real data mutated vs copied sample
- manual step not captured
- bench no warmup/repeats/context
- randomness no seed/variance
- claim from 1 happy path
- broad cleanup:
`rm -rf *`, `git clean -fdx` sans consent