# Correlated Errors in Large Language Models

arXiv: https://arxiv.org/abs/2506.07962 · HTML: https://mlsanity.com/paper/2506.07962/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 49 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 30 of 41)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Large language models that replace human participants can harmfully misportray and flatten identity groups](https://mlsanity.com/paper/2402.01908/) (2024)
- [Algorithmic Monoculture and Social Welfare](https://mlsanity.com/paper/2101.05853/) (2021)
- [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022)
- [Harnessing Multiple Large Language Models: A Survey on LLM Ensemble](https://mlsanity.com/paper/2502.18036/) (2025)
- [Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency](https://mlsanity.com/paper/2006.16736/) (2020)
- [Hypothesis Generation with Large Language Models](https://mlsanity.com/paper/2404.04326/) (2024)
- [Great Models Think Alike and this Undermines AI Oversight](https://mlsanity.com/paper/2502.04313/) (2025)
- [Harnessing the Universal Geometry of Embeddings](https://mlsanity.com/paper/2505.12540/) (2025)
- [Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval](https://mlsanity.com/paper/2407.20371/) (2024)
- [Evaluating LLMs at Detecting Errors in LLM Responses](https://mlsanity.com/paper/2404.03602/) (2024)
- [Echoes in AI: Quantifying lack of plot diversity in LLM outputs](https://mlsanity.com/paper/2501.00273/) (2024)
- [Toward an Evaluation Science for Generative AI Systems](https://mlsanity.com/paper/2503.05336/) (2025)
- [LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity](https://mlsanity.com/paper/2410.03953/) (2024)
- [We're Different, We're the Same: Creative Homogeneity Across LLMs](https://mlsanity.com/paper/2501.19361/) (2025)
- [Ecosystem Graphs: The Social Footprint of Foundation Models](https://mlsanity.com/paper/2303.15772/) (2023)
- [Beyond Aesthetics: Cultural Competence in Text-to-Image Models](https://mlsanity.com/paper/2407.06863/) (2024)
- [Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench](https://mlsanity.com/paper/2407.13696/) (2024)
- [Sparse Autoencoders for Hypothesis Generation](https://mlsanity.com/paper/2502.04382/) (2025)
- [Validating LLM-as-a-Judge Systems under Rating Indeterminacy](https://mlsanity.com/paper/2503.05965/) (2025)
- [Evaluating Generative AI Systems is a Social Science Measurement Challenge](https://mlsanity.com/paper/2411.10939/) (2024)
- [Generative Monoculture in Large Language Models](https://mlsanity.com/paper/2407.02209/) (2024)
- [Auditing the Use of Language Models to Guide Hiring Decisions](https://mlsanity.com/paper/2404.03086/) (2024)

## Built on by (most cited first, 30 of 45)

- [When Your AI Agent Succumbs to Peer-Pressure: Studying Opinion-Change Dynamics of LLMs](https://mlsanity.com/paper/2510.19107/) (2025)
- [When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models](https://mlsanity.com/paper/2606.27288/) (2026)
- [The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?](https://mlsanity.com/paper/2601.23045/) (2026)
- [Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness](https://mlsanity.com/paper/2603.06612/) (2026)
- [Safety and accuracy follow different scaling laws in clinical large language models](https://mlsanity.com/paper/2605.04039/) (2026)
- [When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?](https://mlsanity.com/paper/2510.20963/) (2025)
- [Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models](https://mlsanity.com/paper/2601.22336/) (2026)
- [The Subjectivity of Monoculture](https://mlsanity.com/paper/2602.24086/) (2026)
- [Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact](https://mlsanity.com/paper/2603.00883/) (2026)
- [Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation](https://mlsanity.com/paper/2606.07616/) (2026)
- [Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity](https://mlsanity.com/paper/2606.07726/) (2026)
- [Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Models](https://mlsanity.com/paper/2606.16999/) (2026)
- [State-dependent error correlations shape voting thresholds in committees of AI agents](https://mlsanity.com/paper/2607.23931/) (2026)
- [Council Mode: A Heterogeneous Multi-Agent Consensus Framework for Reducing LLM Hallucination and Bias](https://mlsanity.com/paper/2604.02923/) (2026)
- [Pramana: A Protocol-Layer Treatment of Claim Verification in Autonomous Agent Networks](https://mlsanity.com/paper/2605.20312/) (2026)
- [Design and Evaluation of Multi-Agent AI Oracle Systems for Prediction Market Resolution](https://mlsanity.com/paper/2605.30802/) (2026)
- [Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shift](https://mlsanity.com/paper/2606.25152/) (2026)
- [Harnessing Disagreement: Detecting Correlated Agreement Blindness in Multi-Agent Triage](https://mlsanity.com/paper/2607.19899/) (2026)
- [Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs](https://mlsanity.com/paper/2607.20464/) (2026)
- [The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing](https://mlsanity.com/paper/2609.07713/) (2026)
- [Theory of Scene: Breaking the Symmetry Trap in Multi-Agent LLM Coordination](https://mlsanity.com/paper/2609.32939/) (2026)
- [Uncovering Competency Gaps in Large Language Models and Their Benchmarks](https://mlsanity.com/paper/2512.20638/) (2025)
- [Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMs](https://mlsanity.com/paper/2604.02230/) (2026)
- [LACE: Lattice Attention for Cross-thread Exploration](https://mlsanity.com/paper/2604.15529/) (2026)
- [Refute-or-Promote: An Adversarial Stage-Gated Multi-Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery](https://mlsanity.com/paper/2604.19049/) (2026)
- [The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes](https://mlsanity.com/paper/2606.30653/) (2026)
- [How Much Does Correctness Cost? Budgeted Placement of Strong Correctors in a Weak Multi-Agent Swarm](https://mlsanity.com/paper/2607.09765/) (2026)
- [PlanFlip: Attacking Multi-Agent LLM Systems via Planning-Phase Prompt Injection](https://mlsanity.com/paper/2607.16199/) (2026)
- [Sixteen models, fewer than two voices: measuring ensemble dispersion where no answer is uniquely correct](https://mlsanity.com/paper/2608.00285/) (2026)
- [Decomposing Wrong-Consensus Agreement in LLM Self-Consistency](https://mlsanity.com/paper/2608.18795/) (2026)

## Abstract (arXiv)

Diversity in training data, architecture, and providers is assumed to mitigate homogeneity in LLMs. However, we lack empirical evidence on whether different LLMs differ meaningfully. We conduct a large-scale empirical evaluation on over 350 LLMs overall, using two popular leaderboards and a resume-screening task. We find substantial correlation in model errors -- on one leaderboard dataset, models agree 60% of the time when both models err. We identify factors driving model correlation, including shared architectures and providers. Crucially, however, larger and more accurate models have highly correlated errors, even with distinct architectures and providers. Finally, we show the effects of correlation in two downstream tasks: LLM-as-judge evaluation and hiring -- the latter reflecting theoretical predictions regarding algorithmic monoculture.