Research checked 2026-11-06.
Round table here means independent reviews plus evidence-led adjudication, not mandatory interactive debate.
Two judges is a bounded engineering default, not a demonstrated optimum for PiPi workflows.
Different model families do not eliminate shared errors, especially when judges receive the same incomplete evidence.
Scores remain conditional judgments, not calibrated probabilities or measurements of unobserved benefits.
Findings
PoLL, 2024 evaluated heterogeneous panels across six datasets and reported better performance and over sevenfold lower cost than its single-large-judge baseline.
Its three-judge setup does not establish two judges, diversity, or smaller models as universally optimal.
ContextualJudgeBench, ACL 2025 found contextual assessment difficult; its jury and self-consistency experiments rarely improved on the strongest judge.
More calls do not guarantee better contextual judgments.
Beyond Consensus, EMNLP 2026 found consensus debate could reduce human alignment through asymmetric role framing and strict-stance dominance.
Its score-masking ablation concerns interactive debate, not independent first reviews; it does not establish that hiding peer scores always improves accuracy.
The Coin Flip Judge, GroundLM 2026 found substantial repeated-verdict instability and candidate-order effects in its evaluated tasks and models.
Opposite orders across two initial judges limit common positional exposure, but do not prove order invariance or replace repeated trials when consequential close calls warrant them.
Design boundary
The short skill selects a budgeted review protocol from mixed evidence, not a state-of-the-art accuracy guarantee.
Neutral roles, independent first returns, primary-source checks, explicit unknowns, and preserved dissent avoid treating agreement as proof.
A bounded follow-up can resolve a specific factual dispute; unresolved uncertainty can remain unresolved.
Automatic averaging of ordinal ratings, escalating panel size, or debating until agreement has no universal justification in these sources.
Delegation mechanics, handoff projection, resource scope, and recovery remain owned by pipi-workflows.
# Research limits
Research checked 2026-11-06.
Round table here means independent reviews plus evidence-led adjudication, not mandatory interactive debate.
Two judges is a bounded engineering default, not a demonstrated optimum for PiPi workflows.
Different model families do not eliminate shared errors, especially when judges receive the same incomplete evidence.
Scores remain conditional judgments, not calibrated probabilities or measurements of unobserved benefits.
## Findings
- [PoLL, 2024](https://arxiv.org/abs/2404.18796) evaluated heterogeneous panels across six datasets and reported better performance and over sevenfold lower cost than its single-large-judge baseline.
Its three-judge setup does not establish two judges, diversity, or smaller models as universally optimal.
- [ContextualJudgeBench, ACL 2025](https://aclanthology.org/2025.acl-long.470/) found contextual assessment difficult; its jury and self-consistency experiments rarely improved on the strongest judge.
More calls do not guarantee better contextual judgments.
- [Beyond Consensus, EMNLP 2026](https://aclanthology.org/2026.findings-emnlp.243/) found consensus debate could reduce human alignment through asymmetric role framing and strict-stance dominance.
Its score-masking ablation concerns interactive debate, not independent first reviews; it does not establish that hiding peer scores always improves accuracy.
- [The Coin Flip Judge, GroundLM 2026](https://aclanthology.org/2026.groundlm-1.64/) found substantial repeated-verdict instability and candidate-order effects in its evaluated tasks and models.
Opposite orders across two initial judges limit common positional exposure, but do not prove order invariance or replace repeated trials when consequential close calls warrant them.
## Design boundary
The short skill selects a budgeted review protocol from mixed evidence, not a state-of-the-art accuracy guarantee.
Neutral roles, independent first returns, primary-source checks, explicit unknowns, and preserved dissent avoid treating agreement as proof.
A bounded follow-up can resolve a specific factual dispute; unresolved uncertainty can remain unresolved.
Automatic averaging of ordinal ratings, escalating panel size, or debating until agreement has no universal justification in these sources.
Delegation mechanics, handoff projection, resource scope, and recovery remain owned by pipi-workflows.