mlsanityData to 7 Oct 2026

Correlated Errors in Large Language Models

arXiv 2506.07962 ↗First posted 9 Jun 2025cs.CL cs.AI cs.CYcited by 42

Not on the map. Why?

Abstract (arXiv)

Diversity in training data, architecture, and providers is assumed to mitigate homogeneity in LLMs. However, we lack empirical evidence on whether different LLMs differ meaningfully. We conduct a large-scale empirical evaluation on over 350 LLMs overall, using two popular leaderboards and a resume-screening task. We find substantial correlation in model errors -- on one leaderboard dataset, models agree 60% of the time when both models err. We identify factors driving model correlation, including shared architectures and providers. Crucially, however, larger and more accurate models have highly correlated errors, even with distinct architectures and providers. Finally, we show the effects of correlation in two downstream tasks: LLM-as-judge evaluation and hiring -- the latter reflecting theoretical predictions regarding algorithmic monoculture.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 41 of its 49 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

41 of 49 references.

References · 41

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2024A Survey on LLM-as-a-Judgecited by 1,384
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  5. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  6. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  7. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  8. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  9. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 137
  10. 2021Algorithmic Monoculture and Social Welfarecited by 98
  11. 2022Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?cited by 89
  12. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
29 more
  1. 2020Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistencycited by 77
  2. 2024Hypothesis Generation with Large Language Modelscited by 68
  3. 2025Harnessing the Universal Geometry of Embeddingscited by 54
  4. 2025Great Models Think Alike and this Undermines AI Oversightcited by 53
  5. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 49
  6. 2024Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrievalcited by 49
  7. 2024Echoes in AI: Quantifying lack of plot diversity in LLM outputscited by 47
  8. 2024LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversitycited by 42
  9. 2025Toward an Evaluation Science for Generative AI Systemscited by 42
  10. 2025We're Different, We're the Same: Creative Homogeneity Across LLMscited by 41
  11. 2023Ecosystem Graphs: The Social Footprint of Foundation Modelscited by 31
  12. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  13. 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 25
  14. 2025Sparse Autoencoders for Hypothesis Generationcited by 24
  15. 2025Validating LLM-as-a-Judge Systems under Rating Indeterminacycited by 23
  16. 2024Evaluating Generative AI Systems is a Social Science Measurement Challengecited by 22
  17. 2024Generative Monoculture in Large Language Modelscited by 21
  18. 2024Auditing the Use of Language Models to Guide Hiring Decisionscited by 19
  19. 2024Auditing Work: Exploring the New York City algorithmic bias audit regimecited by 18
  20. 2023Algorithmic Pluralism: A Structural Approach To Equal Opportunitycited by 13
  21. 2025Layers at Similar Depths Generate Similar Activations Across LLM Architecturescited by 11
  22. 2023Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomescited by 10
  23. 2024Competition and Diversity in Generative AIcited by 10
  24. 2024Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountabilitycited by 9
  25. 2024Scarce Resource Allocations That Rely On Machine Learning Should Be Randomizedcited by 7
  26. 2023Monoculture in Matching Marketscited by 5
  27. 2024Law and the Emerging Political Economy of Algorithmic Auditscited by 5
  28. 2024Wisdom and Foolishness of Noisy Matching Marketscited by 2
  29. 2025Strategic Hiring under Algorithmic Monoculturecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 39.

42 papers.

Sort:

By area:

  1. 2025When Your AI Agent Succumbs to Peer-Pressure: Studying Opinion-Change Dynamics of LLMscited by 14
  2. 2026When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Modelscited by 10
  3. 2026The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?cited by 8
  4. 2026Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulnesscited by 8
  5. 2026Safety and accuracy follow different scaling laws in clinical large language modelscited by 5
  6. 2025When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?cited by 4
  7. 2026Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impactcited by 2
  8. 2026State-dependent error correlations shape voting thresholds in committees of AI agentscited by 2
  9. 2026Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Modelscited by 1
  10. 2026The Subjectivity of Monoculturecited by 1
  11. 2026Council Mode: A Heterogeneous Multi-Agent Consensus Framework for Reducing LLM Hallucination and Biascited by 1
  12. 2026Pramana: A Protocol-Layer Treatment of Claim Verification in Autonomous Agent Networkscited by 1
30 more
  1. 2026Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similaritycited by 1
  2. 2026Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Modelscited by 1
  3. 2026Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shiftcited by 1
  4. 2026Harnessing Disagreement: Detecting Correlated Agreement Blindness in Multi-Agent Triagecited by 1
  5. 2026Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMscited by 1
  6. 2026The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishingcited by 1
  7. 2025Uncovering Competency Gaps in Large Language Models and Their Benchmarkscited by 0
  8. 2026Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMscited by 0
  9. 2026LACE: Lattice Attention for Cross-thread Explorationcited by 0
  10. 2026Refute-or-Promote: An Adversarial Stage-Gated Multi-Agent Review Methodology for High-Precision LLM-Assisted Defect Discoverycited by 0
  11. 2026Design and Evaluation of Multi-Agent AI Oracle Systems for Prediction Market Resolutioncited by 0
  12. 2026Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimationcited by 0
  13. 2026The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakescited by 0
  14. 2026How Much Does Correctness Cost? Budgeted Placement of Strong Correctors in a Weak Multi-Agent Swarmcited by 0
  15. 2026PlanFlip: Attacking Multi-Agent LLM Systems via Planning-Phase Prompt Injectioncited by 0
  16. 2026Sixteen models, fewer than two voices: measuring ensemble dispersion where no answer is uniquely correctcited by 0
  17. 2026Decomposing Wrong-Consensus Agreement in LLM Self-Consistencycited by 0
  18. 2026Where Reliability Lives: Experimental Localisation of Behavioural Properties in an Agent Systemcited by 0
  19. 2026Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Modelscited by 0
  20. 2026Why Better Models Can Create Riskier Systems: Evidence from LLM Agents in Financial Marketscited by 0
  21. 2026Monocultural Biases: Correlated biases in large language models lead to unequal systemic exclusion rates in hiringcited by 0
  22. 2026Unanimity Without Persuasion: A Single Round of Debate Erases the Disagreement That Verification Needscited by 0
  23. 2026How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?cited by 0
  24. 2026Multi-agent Scaling Across Disjunctive and Compensatory Taskscited by 0
  25. 2026Theory of Scene: Breaking the Symmetry Trap in Multi-Agent LLM Coordinationcited by 0
  26. 2026Social Circuits behind Multi-agent Echo Chamberscited by 0
  27. 2026On the Limits of Metacognitive Monitoring in LLMscited by 0
  28. 2026VStress: Correlation-Aware Auditing and Adaptive Budget Allocation for Repeated Verifierscited by 0
  29. 2026RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detectioncited by 0
  30. 2026Self-Repulsive Sampling for Diffusion Language Modelscited by 0