Prompt keywords ≠ instruction-following; an instruction and its negation can contain identical keywords.
Test delivery, routing, data handling, and privacy separately.
Mocked replies verify plumbing, not real model/provider behavior.
Evaluate actual outputs against task intent and representative failure scenarios, including conflicting evidence and permissions where relevant.
Criteria must not depend on prompt phrasing.
Record actual model, observed outputs, outcomes, failures, limits.
Use existing evaluation infrastructure.
Keep costly or variable evaluations outside the default fast loop unless required.
Evaluation unavailable or disproportionate → review the change and disclose the gap.
No keyword/regex/snapshot substitute for behavioral evidence.
Retain reusable scenarios; consolidate overlap, discard scaffolding without hiding failures.
# Prompt / model evaluation
- Prompt keywords ≠ instruction-following; an instruction and its negation can contain identical keywords.
- Test delivery, routing, data handling, and privacy separately.
Mocked replies verify plumbing, not real model/provider behavior.
- Evaluate actual outputs against task intent and representative failure scenarios, including conflicting evidence and permissions where relevant.
Criteria must not depend on prompt phrasing.
- Record actual model, observed outputs, outcomes, failures, limits.
- Use existing evaluation infrastructure.
Keep costly or variable evaluations outside the default fast loop unless required.
- Evaluation unavailable or disproportionate → review the change and disclose the gap.
No keyword/regex/snapshot substitute for behavioral evidence.
- Retain reusable scenarios; consolidate overlap, discard scaffolding without hiding failures.