3 +description: "Use when benchmarking command-line performance: comparing commands, measuring CLI/runtime changes, exporting structured results, or designing reproducible microbenchmarks."
8 +Use `hyperfine` for command-line benchmarks, not correctness tests.
12 +Check current local behavior first:
20 +Then consult upstream README only for examples/background.
22 +##
Minimal useful benchmark 25 +hyperfine --style basic --warmup 3 --min-runs 10 \
26 + --export-json bench.json \
27 + --export-markdown bench.md \
28 + --command-name
'before' 'cmd before' \
29 + --command-name
'after' 'cmd after' 34 +1. Benchmark comparable commands with identical inputs, env, cwd, caches, and data.
35 +2. Include a baseline/control; one command alone rarely answers anything.
36 +3. Name the measured workload: cold start, startup+I/O, hot loop, throughput, latency, etc.
37 +4. Use warmups for cache/JIT/startup effects; use `--prepare` for per-run reset.
38 +5. Lifecycle order is: `--setup` once, then for warmups and measured runs `--prepare` → command → `--conclude`, then `--cleanup` once.
39 +6. Avoid sub-5 ms shell noise; prefer `-N`/`--shell=none` for simple direct commands.
40 +7. Do not benchmark interactive flows; make commands non-interactive first.
41 +8. Non-zero exit stops the benchmark; use `--ignore-failure` or `--ignore-failure=CODE` only when failure is expected data.
43 +##
Stdio, output, and logs 47 +- stdin is `/dev/null`; use `--input FILE` for deterministic input.
48 +- stdout/stderr are suppressed; use `--show-output` only for debugging.
49 +- `--show-output` may conflict with `--style`; drop `--style basic` while debugging output.
50 +- `--output null` is default and may trigger program optimizations.
51 +- Use `--output pipe` when a program behaves differently with `/dev/null`.
52 +- Use `--output inherit` only when intentionally measuring terminal output.
53 +- Use `--output ./file`, not `--output file`; a file target needs a path.
54 +- `--output ./file` is overwritten each run; it is not a per-run log.
55 +- To keep per-run logs, redirect inside the command and include `$HYPERFINE_ITERATION` in the filename.
57 +`$HYPERFINE_ITERATION` is available inside the benchmarked command: `warmup-0`, then `0`, `1`, … for measured runs. Do not rely on it in `--prepare`/`--conclude`.
62 +hyperfine --output pipe
'my-cmd < input.txt > "logs/run-${HYPERFINE_ITERATION}.log"' 67 +- Default Unix shell is `/bin/sh`; shell syntax, globs, pipes, redirects, and `&&` need a shell.
68 +- Use `--shell bash --norc` only for Bash-specific syntax/functions.
69 +- Use `-N`/`--shell=none` for direct executable benchmarks; no shell expansion, pipes, redirects, variables, globs, or `&&` syntax occurs.
70 +- With `-N`, shell syntax becomes literal argv; it may silently benchmark the wrong command.
71 +- With `-N`, quote any single argv containing spaces inside the command string, e.g. `"+lua dofile('x.lua')"`.
72 +- Quote commands as single strings so hyperfine receives each benchmark command intact.
76 +Use built-in parameterization instead of hand-written loops:
79 +hyperfine -P threads 1 8
'make -j {threads}' 80 +hyperfine -P delay 0.3 0.7 -D 0.2
'sleep {delay}' 81 +hyperfine -L mode fast,safe
'./tool --mode {mode}' 84 +Multiple `-L` values produce a Cartesian product; inspect JSON or stdout to verify generated commands.
86 +##
Workload-specific controls 88 +- Align semantics first: traversal scope, matching mode, ignore rules, formatting, color, paging, config, and output shape.
89 +- Separate startup, parsing/compilation, rendering, I/O, and steady-state work with no-op and feature-off controls.
90 +- Label what was measured; do not turn a cold-start or UI-rendering result into a throughput claim.
91 +- Simple serial tools can beat richer/parallel tools on small or trivial workloads.
93 +##
VM, interpreter, and JIT benchmarks 95 +Cold-start scripts mostly measure process startup, dynamic linking, parser/init cost, and trivial I/O—not peak VM speed.
97 +Controls for VM claims:
99 +- Verify output once with `--show-output` or direct runs before timing.
100 +- Record exact versions, resolved executables, package/build source, binary size, and dynamic libraries (`command -v`, `readlink -f`, `--version`, `ldd`).
101 +- Add a no-op/empty-script control to separate startup from workload.
102 +- Add feature-off controls when available, e.g. JIT disabled, optimizer disabled, config disabled.
103 +- Add a hot-workload control when comparing JIT VMs; trivial scripts may never trigger compilation.
104 +- Report the result as “startup”, “startup+script”, or “hot loop”; never generalize cold-start results to language/VM throughput.
106 +##
Structured output 108 +Hyperfine supports structured exports:
110 +- `--export-json FILE`: summary plus individual run timings, seconds.
111 +- `--export-csv FILE`: summary only, seconds.
112 +- `--export-markdown FILE`: user-ready table.
113 +- `--export-asciidoc FILE`, `--export-orgmode FILE`: docs tables.
115 +`--reference`/`--reference-name` improves the stdout relative comparison. Verify exported table semantics locally before claiming a custom reference; Markdown may still present relatives against the fastest command.
117 +JSON fields vary by version; inspect with:
120 +jq
'.results[0] | keys' bench.json
123 +Common fields: `command`, `mean`, `stddev`, `median`, `min`, `max`, `times`, `user`, `system`, `exit_codes`.
125 +##
Presenting results 127 +Show the Markdown table when possible, then one conclusion sentence. If `--reference` matters, include the stdout “Relative speed comparison” too.
131 +- command names, not giant command strings; use `--command-name`.
132 +- mean ± stddev, min/max, relative speed.
133 +- run count, warmup count, hyperfine version.
134 +- benchmark context: machine/load, input size, cache state, cwd, relevant env.
135 +- caveat when hyperfine warns about very fast commands or outliers.
137 +Do not overclaim. Say “in this run/context” unless results were repeated independently.
141 +- Never print secrets from commands, env, logs, JSON, or Markdown exports.
142 +- Do not run `sudo`, cache-dropping, destructive cleanup, network writes, or production mutations without explicit approval.
143 +- Keep artifacts in a temp or project benchmark dir; delete scratch logs unless promoted.