mlsanityData to 14 Oct 2026

Decomposing Wrong-Consensus Agreement in LLM Self-Consistency

arXiv 2608.18795 ↗First posted 19 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Agreement among repeated samples of a language model is routinely read as evidence about answer reliability, yet wrong answers can agree just as strongly as right ones. This paper asks what information wrong-consensus agreement actually contains, and answers with a quantitative decomposition. A pluralistic agreement index Gamma, normalized by the reference scale d=(1-p)/(C-1), is split into a mechanical component (agreement delivered by a per-case answer preference alone) and a preference-unexplained residual. The mechanical reference is leak-free: each case's preference and accuracy are estimated from its other runs only. On public GPT-4.1 per-run data, coverage phi (the mechanical/empirical ratio) shows a benchmark-associated direction: 0.81-0.93 on multiple-choice GPQA-Diamond against 0.59-0.78 on open-domain AIME, where a residual of 1.54-2.80 Gamma units survives, more than absorbed by a calibrated run-level preference-heterogeneity reference. A controlled replication under one fixed protocol (four runs per question, K=32 votes) on five open-weights checkpoints (Qwen3.5-9B/122B, Qwen3.8-27B, Gemma4-26B/31B) finds near-complete mechanical coverage in all ten cells (phi approximately 1, with a small overshoot consistent with a quantified finite-donor plug-in bias), robust to a two-run design; the largest cell (qwen3.5-122b, p=0.222) sits inside the GPT-4.1 AIME accuracy range and still saturates (phi=1.041). A cross-system contrast at comparable aggregate accuracy contrasts near-complete mechanical agreement in the open-weights models against a larger preference-unexplained residual in the frontier family. This contrast is confounded with sampling protocol by design. Agreement is graded evidence, not certification. No new voting method is proposed; code and evidence are committed.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 28 references that are in the map. First posted on arXiv 19 Aug 2026.

Builds on

15 of 28 references.

References · 15

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  3. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  4. 2025Correlated Errors in Large Language Modelscited by 45
  5. 2025Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similaritycited by 28
  6. 2025Too Consistent to Detect: A Study of Self-Consistent Errors in LLMscited by 18
  7. 2025Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMscited by 12
  8. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 10
  9. 2026Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensemblescited by 9
  10. 2025The Consistency Hypothesis in Uncertainty Quantification for Large Language Modelscited by 6
  11. 2026Beyond Consensus: Trace-Level Synthesis in Mixture of Agentscited by 4
  12. 2026When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMscited by 3
3 more
  1. 2026When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signalscited by 2
  2. 2026Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMscited by 1
  3. 2026AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficultycited by 1

Built on by

Not cited yet.