# Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles

arXiv: https://arxiv.org/abs/2607.20768 · HTML: https://mlsanity.com/paper/2607.20768/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 22 Jul 2026.

## Builds on (most cited first, 19 of 19)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [A Unified Theory of Diversity in Ensemble Learning](https://mlsanity.com/paper/2301.03962/) (2023)
- [Great Models Think Alike and this Undermines AI Oversight](https://mlsanity.com/paper/2502.04313/) (2025)
- [Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?](https://mlsanity.com/paper/2502.00674/) (2025)
- [Getting MoRE out of Mixture of Language Model Reasoning Experts](https://mlsanity.com/paper/2305.14628/) (2023)
- [LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity](https://mlsanity.com/paper/2410.03953/) (2024)
- [Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information](https://mlsanity.com/paper/2510.01499/) (2025)
- [When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models](https://mlsanity.com/paper/2606.27288/) (2026)
- [Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection](https://mlsanity.com/paper/2602.08003/) (2026)
- [Quantifying Diversity of Thought: A Predictive Law of Weighted LLM Ensemble Lift](https://mlsanity.com/paper/2607.17384/) (2026)
- [Examining Independence in Ensemble Sentiment Analysis: A Study on the Limits of Large Language Models Using the Condorcet Jury Theorem](https://mlsanity.com/paper/2409.00094/) (2024)
- [DFPE: A Diverse Fingerprint Ensemble for Enhancing LLM Performance](https://mlsanity.com/paper/2501.17479/) (2025)
- [LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition](https://mlsanity.com/paper/2607.13347/) (2026)

## Built on by (most cited first, 9 of 9)

- [When LLMs Sit Above Diagnostic Tools: Unrealized Complementarity in Industrial Fault Diagnosis](https://mlsanity.com/paper/2610.05031/) (2026)
- [Sixteen models, fewer than two voices: measuring ensemble dispersion where no answer is uniquely correct](https://mlsanity.com/paper/2608.00285/) (2026)
- [Decorrelation Is Not Complementarity: Skill, Not Lineage, Governs Trusted-Monitor Ensembles](https://mlsanity.com/paper/2608.16190/) (2026)
- [Decomposing Wrong-Consensus Agreement in LLM Self-Consistency](https://mlsanity.com/paper/2608.18795/) (2026)
- [A latent dimension of Condorcet's jury theorem for multiple AI advisers](https://mlsanity.com/paper/2609.14438/) (2026)
- [Feed the Panel Dimensions, Not Verdicts: Rubric-Decomposed Fusion of Vision-Language Aesthetic Judges](https://mlsanity.com/paper/2609.27110/) (2026)
- [Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?](https://mlsanity.com/paper/2609.32996/) (2026)
- [RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection](https://mlsanity.com/paper/2609.39229/) (2026)
- [Where Do LLM Ensemble Gains Come From? Decomposing Candidate Value, Diversity, and Aggregation](https://mlsanity.com/paper/2610.18786/) (2026)

## Abstract (arXiv)

Majority voting over LLMs is widely assumed to benefit from diversity, and diversity measures are used to choose which models to combine. We ask whether five such measures track diversity or mainly re-express capability, auditing them as predictors of majority-vote gain over the best member across 31,900 subsets of 30 LLMs on MMLU-Pro (29 on TruthfulQA) under explicit capability controls. Three findings emerge. First, latent complementarity is ubiquitous: oracle gain is positive in 100% of subsets, yet simple voting beats the strongest member in only 9.98% of all canonical size-3 subsets (18.71% with held-out best selection); the pooled size-2-4 rate is 1.27%, partly reflecting deterministic even-size voting behavior. Second, a joint-correctness proxy (strict diversity) is nearly collinear with one minus mean accuracy (size-3 Spearman rho = +0.991 / +0.988); raw diversity-gain associations are strongly capability-entangled and, with one exception, unstable under control. Third, three linear contingency-table statistics are algebraically non-separable; after capability control, the empirically stable remainder is a modest residual pairwise co-failure association in which more shared error corresponds to lower gain. This direction is robust, but its magnitude is configuration-dependent. Joint rawspace linear regressions treating strict diversity, disagreement, and double-fault as independent predictors are rank-deficient by construction.