# Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

arXiv: https://arxiv.org/abs/2510.22954 · HTML: https://mlsanity.com/paper/2510.22954/

## Key facts

- Cited by 141 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2025.

## Built on by (most cited first, 30 of 141)

- [Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels](https://mlsanity.com/paper/2605.29800/) (2026)
- [Stop Automating Peer Review Without Rigorous Evaluation](https://mlsanity.com/paper/2605.03202/) (2026)
- [OdysSim: Building Foundation Models for Human Behavior Simulation](https://mlsanity.com/paper/2606.14199/) (2026)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [Evaluating LLMs in Finance Requires Explicit Bias Consideration](https://mlsanity.com/paper/2602.14233/) (2026)
- [Where does output diversity collapse in post-training?](https://mlsanity.com/paper/2604.16027/) (2026)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges](https://mlsanity.com/paper/2603.11863/) (2026)
- [StoryScope: Investigating idiosyncrasies in AI fiction](https://mlsanity.com/paper/2604.03136/) (2026)
- [Agentic AI Scientists Are Not Built For Autonomous Scientific Discovery](https://mlsanity.com/paper/2605.08956/) (2026)
- [CREATE: Testing LLMs for Associative Creativity](https://mlsanity.com/paper/2603.09970/) (2026)
- [Generative AI collective behavior needs an interactionist paradigm](https://mlsanity.com/paper/2601.10567/) (2026)
- [MoCo: A One-Stop Shop for Model Collaboration Research](https://mlsanity.com/paper/2601.21257/) (2026)
- [Unlocking LLM Creativity in Science through Analogical Reasoning](https://mlsanity.com/paper/2605.11258/) (2026)
- [Attractor States Emerge in Multi-Turn LLM Conversations](https://mlsanity.com/paper/2606.30571/) (2026)
- [One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL](https://mlsanity.com/paper/2608.12253/) (2026)
- [AgenticRed: Evolving Agentic Systems for Red-Teaming](https://mlsanity.com/paper/2601.13518/) (2026)
- [Annotations Mitigate Post-Training Mode Collapse](https://mlsanity.com/paper/2605.09995/) (2026)
- [Language of Thought Shapes Output Diversity in Large Language Models](https://mlsanity.com/paper/2601.11227/) (2026)
- [ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review](https://mlsanity.com/paper/2601.22638/) (2026)
- [FAMOSE: A ReAct Approach to Automated Feature Discovery](https://mlsanity.com/paper/2602.17641/) (2026)
- [Auditing Preferences for Brands and Cultures in LLMs](https://mlsanity.com/paper/2603.18300/) (2026)
- [Reinforcing Human Behavior Simulation via Verbal Feedback](https://mlsanity.com/paper/2605.20506/) (2026)
- [EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World](https://mlsanity.com/paper/2601.01530/) (2026)
- [Markovian Generation Chains in Large Language Models](https://mlsanity.com/paper/2603.11228/) (2026)
- [The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices](https://mlsanity.com/paper/2603.18482/) (2026)
- [AI generates well-liked but templatic empathic responses](https://mlsanity.com/paper/2604.08479/) (2026)
- [What Do Evolutionary Coding Agents Evolve?](https://mlsanity.com/paper/2605.20086/) (2026)
- [The One-Word Census: Answer-Choice Conformity Across 105 Language Models](https://mlsanity.com/paper/2607.12796/) (2026)
- [The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisions](https://mlsanity.com/paper/2608.05583/) (2026)

## Abstract (arXiv)

Language models (LMs) often struggle to generate diverse, human-like creative content, raising concerns about the long-term homogenization of human thought through repeated exposure to similar outputs. Yet scalable methods for evaluating LM output diversity remain limited, especially beyond narrow tasks such as random number or name generation, or beyond repeated sampling from a single model. We introduce Infinity-Chat, a large-scale dataset of 26K diverse, real-world, open-ended user queries that admit a wide range of plausible answers with no single ground truth. We introduce the first comprehensive taxonomy for characterizing the full spectrum of open-ended prompts posed to LMs, comprising 6 top-level categories (e.g., brainstorm & ideation) that further breaks down to 17 subcategories. Using Infinity-Chat, we present a large-scale study of mode collapse in LMs, revealing a pronounced Artificial Hivemind effect in open-ended generation of LMs, characterized by (1) intra-model repetition, where a single model consistently generates similar responses, and more so (2) inter-model homogeneity, where different models produce strikingly similar outputs. Infinity-Chat also includes 31,250 human annotations, across absolute ratings and pairwise preferences, with 25 independent human annotations per example. This enables studying collective and individual-specific human preferences in response to open-ended queries. Our findings show that LMs, reward models, and LM judges are less well calibrated to human ratings on model generations that elicit differing idiosyncratic annotator preferences, despite maintaining comparable overall quality. Overall, INFINITY-CHAT presents the first large-scale resource for systematically studying real-world open-ended queries to LMs, revealing critical insights to guide future research for mitigating long-term AI safety risks posed by the Artificial Hivemind.