9 9 ##
Commands / settings 11 11 - `/iq` opens the responsive leaderboard.
12 -- `/iq scan` estimates and confirms judge cost before scoring closed sessions.
12 +- `/iq scan [n]` estimates and confirms judge cost before scoring closed sessions, at most `n`.
13 +- `/iq counter [n]` re-scores already scored sessions with the counter judges to measure judge disagreement.
14 +- `/iq check <provider/model>` runs an unseen model through the experimental scripted-task check; the dashboard offers the same via `c`.
13 15 - `iq.judgeModel` selects `provider/model`; `null` or omission uses Pi's configured default.
16 +- `iq.counterJudgeModel` selects the counter judge, as one `provider/model` or a list of them; omission uses the current session model.
17 +- `iq.counterSampleRate` (0 to 1) also counter-judges that share of each scan, sampled from session content so the sample is identical on every host and run.
18 + Unset plus configured `iq.counterJudgeModel` samples a default 0.2 share (configuring judges declares intent to use them); an explicit `0` disables sampling.
19 +- `iq.stateDir` moves all IQ state (records, telemetry, checks) into a shared, synced directory; unset keeps the per-host platform state directory.
17 23 IQ discovers sessions without model calls, then scores only after explicit confirmation.
18 -Results live outside sessions and workspaces in the platform state directory.
24 +Results live outside sessions and workspaces in the platform state directory, or in `iq.stateDir` when set.
19 25 Mixed-model and trivial sessions remain visible but unranked.
27 +Scores are computed from stored observations at read time, never from a frozen number, so the arithmetic can change without re-judging anything.
28 +A dimension score is `100 + 15 × (weighted mean rating − the judge's own mean rating)`, so 100 means typical and judge leniency cancels out.
29 +Thin evidence shrinks toward that same mean, because a fixed anchor plus a lenient judge would score long sessions higher regardless of quality.
31 +Judges often miss the strict tool schema, so malformed observations are repaired or dropped rather than discarding the chunk.
32 +Only cited, in-chunk evidence survives, and repair counts are recorded next to retries.
33 +Failed sessions report why they failed instead of a bare count.
35 +Telemetry is measured while reading sessions and needs no model call.
36 +It lives beside records so its schema can change without invalidating paid judgments.
37 +Provider-reported cost is ignored for ranking, because subscription plans report zero.
39 +Judges agree on model ordering but not on individual sessions, so a counter judge is calibration, not redundancy.
40 +Per-model bias in the method view is where a judge scoring its own family higher becomes visible.
42 +Several counter judges form a rotation pool rather than a panel: each sampled session goes to exactly one of them, assigned from session content.
43 +Adding counter judges therefore buys calibration breadth at constant cost, and splits the paired sessions between them.
44 +Each counter is compared against the primary judge separately, so their own leniency never has to match.
46 +Archetypes (`agentic`, `worker`, `texter`) come from session shape, not from the judge.
47 +Fit combines judged quality, measured output speed, and current registry prices, relative to the models present.
48 +A model needs at least three sessions in an archetype before it is rated there.
50 +##
Shared state across hosts 52 +`iq.stateDir` points IQ at one shared, synced directory (for example a Syncthing folder).
53 +Every host writes into the same `records/`, `telemetry/`, and `checks/`; records carry a `host` field for provenance, and the leaderboard merges all hosts.
54 +The first switch into an empty shared directory copies the local state once, so nothing has to be re-scored.
55 +The scan lock lives in the OS temp directory, because a lock file inside a synced folder would block scans on other hosts for no reason.
56 +Merging keys on session content hash, judge, and rubric version, and the newest score wins.
60 +- `leaderboard` ranks models by judged ability, with dimensions and archetypes on selection.
61 +- `sessions` lists judged sessions with their citations.
62 +- `checks` shows each model's latest `/iq check` result: task states, acceptance, estimate, and check-versus-history deltas.
63 +- `telemetry` shows measured behavior per model: turns, tool calls, edit share, output volume, cache share, latency, and speed.
64 +- `method` explains the scoring math and counter-judge agreement.
66 +##
Experimental model check 68 +`/iq check` estimates a model that has no stored sessions yet, using the same dimensions and the same judge.
69 +It runs three scripted tasks (easy, medium, hard) in frozen workspaces under `extensions/iq/check/fixtures/`.
70 +Each task hides a recipe the candidate never sees; items are delivered one at a time through `pi --mode json --session`, so the model works with real tools across a sustained trajectory.
71 +The judge scores each step from the new session entries and the final state from objective acceptance checks plus the repository diff.
72 +Acceptance checks are either short shell commands or Node verifier scripts (`checks/*.mjs`) that live in the fixture directory, outside the workspace, so candidates cannot tamper with their own grading.
73 +Any fixture edit changes its hash and invalidates old runs of that task.
75 +Check runs are stored separately under `checks/` and never enter session counts, archetypes, or the historical baseline; the dashboard's checks tab summarizes them.
76 +Estimated IQ values are anchored by running already-rated models through the same tasks: per dimension, `mean_i(H_i − 15·R_i) + 15·R_target`.
77 +Until reference checks exist, only raw 1-5 ratings are shown; models with history also get a leave-one-out prediction versus their real rating.
79 +Runs have no time, turn, or token limits, because truncation would measure failure states instead of work.
80 +A stall watchdog pauses runs that stop producing output, and Esc cancels while keeping completed tasks.
81 +A task that produced evidence always persists, even when its final evaluation fails; the incomplete reason distinguishes unreadable sessions, candidate no-work, judge failures, and cancellation.
82 +The driver is a scripted user that does not answer questions; tasks are written to be answerable from the workspace alone.