repositories / smith

smith

There are many coding harnesses - but this one is fast

owned by admin

.pi/skills/benchmarking/SKILL.md

Raw

Rendered preview

name: benchmarking description: "Use before claiming or chasing a performance difference, choosing between two implementations, layouts, or allocators, or pinning a performance budget: pick the signal that answers the question, compare fairly, and keep only what is reproducible."

Benchmarking

Measure the question, not the machine.

Pick the signal

question	signal	tool
does this path allocate more	allocation count, bytes	counting allocator in a test
did this change do more work	instructions, est. cycles, cache hits	callgrind via `gungraun`, one shot, deterministic
is A faster than B on this box	wall clock	`criterion` (lib), `hyperfine` (binary), same machine, same session
where does the time go	profile	`perf stat -d`, `perf record` + flamegraph, `callgrind_annotate`
is the type or record too big	size	`size_of` assertion

Wall clock answers "faster here, now"; instructions answer "less work"; only the second travels between machines.

Compare fairly

A/B on one machine, one session, back to back; interleave runs; nothing else running.
Same toolchain, same profile, same inputs; name baselines by change id (--save-baseline <change>), never before.
Warm up; report median and spread; a difference inside the spread is no difference.
Inputs at the sizes the spec names, and one size larger than fits in cache.
black_box inputs and outputs; check the loop was not optimized away.
Pinned-platform instruction counts come from cargo x ci-image perf; never ad-hoc container copies of the repo.

Model floors

Start from representation and data movement required by the hot path, then measure matched unavoidable work on identical input, residency, hardware, toolchain, and metric.
A measured model baseline is not a literal or theoretical optimum, and never establishes a hardware speed limit from instruction count.
Disk → memory → CPU → memory paths differ from compute-dominated work: state which dominates, and match warm or cold cache residency deliberately.
Throughput and latency answer different questions; collect neither with fake wall-clock loops or typed wrapper work that the hot path does not perform.
Setup allocations stay outside the count; return inputs and outputs so drop stays outside it; black-box both sides so the model cannot disappear.
A model may bulk-copy input only when the implementation must write matched output bytes; tree inputs require a minimal read, visit, and payload-write model rather than a flat serialized copy.
Parsing, validation, sorting, checksums, encoding, event construction, and task-specific work belong outside a bare movement model unless unavoidable by that model definition.
Record model precision and limitations; task-dependent expected work can justify a gap, but no factor, including 100×, is policy.

Keep only what reproduces

Commit: instruction budgets with a tolerance band, allocation counts, size assertions.
Never commit wall-clock numbers; put them in a report or a commit message with the machine named.
A budget moves only by a deliberate commit that says why (toolchain, layout, algorithm).
An improvement beyond the band fails too until re-pinned; a pin that drifts down silently was never a pin.

Read the result

Fewer instructions but more time: memory-bound; look at cache misses and layout before code.
Fewer cache misses but more time: unpacking cost ate the win; drop the encoding.
Callgrind counts exclude wall-clock effects (I/O waits, contention, frequency scaling); pair with one wall-clock run before believing a win.
Binary benches include process start-up; that is the measurement for --help, noise for everything else.

Do not

Benchmark in CI for wall clock; CI containers share cores and vary by an order of magnitude.
Add a benchmark without the spec sentence or bound it proves.

Sources

---
name: benchmarking
description: "Use before claiming or chasing a performance difference, choosing between two implementations, layouts, or allocators, or pinning a performance budget: pick the signal that answers the question, compare fairly, and keep only what is reproducible."
---

# Benchmarking

Measure the question, not the machine.

## Pick the signal

| question | signal | tool |
|---|---|---|
| does this path allocate more | allocation count, bytes | counting allocator in a test |
| did this change do more work | instructions, est. cycles, cache hits | callgrind via `gungraun`, one shot, deterministic |
| is A faster than B on this box | wall clock | `criterion` (lib), `hyperfine` (binary), same machine, same session |
| where does the time go | profile | `perf stat -d`, `perf record` + flamegraph, `callgrind_annotate` |
| is the type or record too big | size | `size_of` assertion |

Wall clock answers "faster here, now"; instructions answer "less work"; only the second travels between machines.

## Compare fairly

- A/B on one machine, one session, back to back; interleave runs; nothing else running.
- Same toolchain, same profile, same inputs; name baselines by change id (`--save-baseline <change>`), never `before`.
- Warm up; report median and spread; a difference inside the spread is no difference.
- Inputs at the sizes the spec names, and one size larger than fits in cache.
- `black_box` inputs and outputs; check the loop was not optimized away.
- Pinned-platform instruction counts come from `cargo x ci-image perf`; never ad-hoc container copies of the repo.

## Model floors

- Start from representation and data movement required by the hot path, then measure matched unavoidable work on identical input, residency, hardware, toolchain, and metric.
- A measured model baseline is not a literal or theoretical optimum, and never establishes a hardware speed limit from instruction count.
- Disk → memory → CPU → memory paths differ from compute-dominated work: state which dominates, and match warm or cold cache residency deliberately.
- Throughput and latency answer different questions; collect neither with fake wall-clock loops or typed wrapper work that the hot path does not perform.
- Setup allocations stay outside the count; return inputs and outputs so drop stays outside it; black-box both sides so the model cannot disappear.
- A model may bulk-copy input only when the implementation must write matched output bytes; tree inputs require a minimal read, visit, and payload-write model rather than a flat serialized copy.
- Parsing, validation, sorting, checksums, encoding, event construction, and task-specific work belong outside a bare movement model unless unavoidable by that model definition.
- Record model precision and limitations; task-dependent expected work can justify a gap, but no factor, including 100×, is policy.

## Keep only what reproduces

- Commit: instruction budgets with a tolerance band, allocation counts, size assertions.
- Never commit wall-clock numbers; put them in a report or a commit message with the machine named.
- A budget moves only by a deliberate commit that says why (toolchain, layout, algorithm).
- An improvement beyond the band fails too until re-pinned; a pin that drifts down silently was never a pin.

## Read the result

- Fewer instructions but more time: memory-bound; look at cache misses and layout before code.
- Fewer cache misses but more time: unpacking cost ate the win; drop the encoding.
- Callgrind counts exclude wall-clock effects (I/O waits, contention, frequency scaling); pair with one wall-clock run before believing a win.
- Binary benches include process start-up; that is the measurement for `--help`, noise for everything else.

## Do not

- Benchmark in CI for wall clock; CI containers share cores and vary by an order of magnitude.
- Add a benchmark without the spec sentence or bound it proves.

## Sources

- [gungraun guide](https://gungraun.github.io/gungraun/)
- [criterion user guide](https://bheisler.github.io/criterion.rs/book/)
- [The Rust Performance Book: benchmarking and profiling](https://nnethercote.github.io/perf-book/benchmarking.html)
- [hyperfine](https://github.com/sharkdp/hyperfine)