Luigit
repositories / dotfiles

dotfiles

bugabingas dorkfiles

owned by admin

pi/agent/skills/testing/references/prompt-evaluation.md

Raw
Rendered preview

Prompt / model evaluation

  • Prompt keywords ≠ instruction-following; an instruction and its negation can contain identical keywords.
  • Test delivery, routing, data handling, and privacy separately. Mocked replies verify plumbing, not real model/provider behavior.
  • Evaluate actual outputs against task intent and representative failure scenarios, including conflicting evidence and permissions where relevant. Criteria must not depend on prompt phrasing.
  • Record actual model, observed outputs, outcomes, failures, limits.
  • Use existing evaluation infrastructure. Keep costly or variable evaluations outside the default fast loop unless required.
  • Evaluation unavailable or disproportionate → review the change and disclose the gap. No keyword/regex/snapshot substitute for behavioral evidence.
  • Retain reusable scenarios; consolidate overlap, discard scaffolding without hiding failures.
# Prompt / model evaluation

- Prompt keywords ≠ instruction-following; an instruction and its negation can contain identical keywords.
- Test delivery, routing, data handling, and privacy separately.
  Mocked replies verify plumbing, not real model/provider behavior.
- Evaluate actual outputs against task intent and representative failure scenarios, including conflicting evidence and permissions where relevant.
  Criteria must not depend on prompt phrasing.
- Record actual model, observed outputs, outcomes, failures, limits.
- Use existing evaluation infrastructure.
  Keep costly or variable evaluations outside the default fast loop unless required.
- Evaluation unavailable or disproportionate → review the change and disclose the gap.
  No keyword/regex/snapshot substitute for behavioral evidence.
- Retain reusable scenarios; consolidate overlap, discard scaffolding without hiding failures.