# Wisdom of LLM Crowds: Aggregation and Contamination in Language Model Ensembles

arXiv: https://arxiv.org/abs/2607.18269 · HTML: https://mlsanity.com/paper/2607.18269/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 44 references that are in the map.
- First posted on arXiv 26 May 2026.

## Builds on (most cited first, 12 of 12)

- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Discovering Symbolic Models from Deep Learning with Inductive Biases](https://mlsanity.com/paper/2006.11287/) (2020)
- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [Investigating Data Contamination in Modern Benchmarks for Large Language Models](https://mlsanity.com/paper/2311.09783/) (2023)
- [Thinking Fast and Slow in Large Language Models](https://mlsanity.com/paper/2212.05206/) (2022)
- [Approaching Human-Level Forecasting with Language Models](https://mlsanity.com/paper/2402.18563/) (2024)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [Forecasting Future World Events with Neural Networks](https://mlsanity.com/paper/2206.15474/) (2022)
- [When is a crowd wise?](https://mlsanity.com/paper/1406.7563/) (2014)
- [Wisdom of crowds: much ado about nothing](https://mlsanity.com/paper/2008.01485/) (2020)

## Built on by (most cited first, 2 of 2)

- [LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications](https://mlsanity.com/paper/2608.23058/) (2026)
- [Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts](https://mlsanity.com/paper/2609.33257/) (2026)

## Abstract (arXiv)

The wisdom of crowds -- the finding that aggregating judgments across individuals often outperforms the best individual -- has been extensively studied with human forecasters. Whether the same phenomenon emerges when the ``crowd'' consists of large language models (LLMs) is an open question with both theoretical and practical implications. We elicited probability estimates from 15 LLMs on 254 binary prediction market questions and evaluated classical and learned aggregation methods. Learned aggregators -- a multilayer perceptron and a logistic regression -- outperformed all individual models and classical methods. The logistic regression was found to match the neural network, suggesting that the benefit of learned aggregation derives from learning a linear combination of diverse model outputs rather than from nonlinear interactions. Symbolic regression applied to the neural network's learned mapping recovered a pure model-disagreement signal as the lowest-complexity useful formula on the Pareto frontier, further supporting this interpretation. Training cutoff contamination proved a pervasive confound: the apparent capability gap between frontier cloud models and smaller local models collapsed from 35.8% to 8.9% on a clean subset of questions resolving after all models' training cutoffs, and individual model rankings showed only moderate stability. Even when the prediction market is evaluated at each model's training cutoff, LLMs remained substantially less accurate, indicating a genuine gap in collective information aggregation. These findings suggest that LLM crowds can exhibit wisdom-of-crowds effects, but that contamination-free evaluation is essential for reliable assessment.