Aggregate Disambiguation Systems
Not on the map. Why?
Abstract (arXiv)
Natural-language tasks can elicit different verdicts from protocol-following evaluators that receive the same declared information. We study aggregate disambiguation systems (ADSs). Given a task and a candidate solution, each evaluator casts a binary vote on whether the solution should be accepted, and the system aggregates the votes of a finite panel. The target is protocol reproducibility relative to an explicitly declared evaluator reference, not semantic truth. We separate fixed finite censuses, probabilistic evaluator populations, and growing-census limits, since their endpoint laws and guarantees are not interchangeable. In the population setting, we use finite samples to estimate how often a finite panel reaches the same decision as the declared evaluator population. We provide a lower confidence bound on the fraction of candidate solutions for which the disagreement probability is at most a chosen tolerance. The calculation accounts separately for sampling candidate solutions and sampling evaluators. The construction permits arbitrary dependence among columns induced by shared evaluator rows and uses exact binomial intervals at the evaluator layer and an exact one-sided binomial inversion at the generator layer. Simulations check the implementation against known population coverages and expose power limitations.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 12 of its 42 references that are in the map. First posted on arXiv 31 Aug 2026.
Builds on
12 of 42 references.
References · 12
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
- 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
- 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 26
- 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 14
- 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judgescited by 5
- 2026A Finite-Calibration Regime Map for LLM Judge Panelscited by 2
- 2026RoPoLL: Robust Panel of LLM Judgescited by 2
- 2016Non-determinism in Byzantine Fault-Tolerant Replicationcited by 1
Built on by
Not cited yet.