# The Variance Paradox: How AI Reduces Diversity but Increases Novelty

arXiv: https://arxiv.org/abs/2508.19264 · HTML: https://mlsanity.com/paper/2508.19264/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 172 references that are in the map.
- First posted on arXiv 20 Aug 2025.

## Builds on (most cited first, 30 of 43)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Fine-tuning language models to find agreement among humans with diverse preferences](https://mlsanity.com/paper/2211.15006/) (2022)
- [Generative artificial intelligence enhances creativity but reduces the diversity of novel content](https://mlsanity.com/paper/2312.00506/) (2023)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [Homogenization Effects of Large Language Models on Human Creative Ideation](https://mlsanity.com/paper/2402.01536/) (2024)
- [Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?](https://mlsanity.com/paper/2211.13972/) (2022)
- [Extractive Opinion Summarization in Quantized Transformer Spaces](https://mlsanity.com/paper/2012.04443/) (2020)
- [Machine Culture](https://mlsanity.com/paper/2311.11388/) (2023)
- [Designing for Responsible Trust in AI Systems: A Communication Perspective](https://mlsanity.com/paper/2204.13828/) (2022)
- [Opportunities and Risks of LLMs for Scalable Deliberation with Polis](https://mlsanity.com/paper/2306.11932/) (2023)
- [Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluation](https://mlsanity.com/paper/2409.14634/) (2024)
- [What is it like to program with artificial intelligence?](https://mlsanity.com/paper/2208.06213/) (2022)
- [OpinionDigest: A Simple Framework for Opinion Summarization](https://mlsanity.com/paper/2005.01901/) (2020)
- [CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AI](https://mlsanity.com/paper/2312.11949/) (2023)
- [The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Models](https://mlsanity.com/paper/2502.11266/) (2025)
- [Superhuman Artificial Intelligence Can Improve Human Decision Making by Increasing Novelty](https://mlsanity.com/paper/2303.07462/) (2023)
- [Generative Monoculture in Large Language Models](https://mlsanity.com/paper/2407.02209/) (2024)
- [AI Scientists Fail Without Strong Implementation Capability](https://mlsanity.com/paper/2506.01372/) (2025)
- [LLMs can Realize Combinatorial Creativity: Generating Creative Ideas via LLMs for Scientific Research](https://mlsanity.com/paper/2412.14141/) (2024)
- [Exploring Perspectives on the Impact of Artificial Intelligence on the Creativity of Knowledge Work: Beyond Mechanised Plagiarism and Stochastic Parrots](https://mlsanity.com/paper/2307.10751/) (2023)
- [The Lock-in Hypothesis: Stagnation by Algorithm](https://mlsanity.com/paper/2506.06166/) (2025)
- [Human-AI Interactions and Societal Pitfalls](https://mlsanity.com/paper/2309.10448/) (2023)
- [Capturing Humans' Mental Models of AI: An Item Response Theory Approach](https://mlsanity.com/paper/2305.09064/) (2023)
- [Epistemic Integrity in Large Language Models](https://mlsanity.com/paper/2411.06528/) (2024)
- [Steering AI-Driven Personalization of Scientific Text for General Audiences](https://mlsanity.com/paper/2411.09969/) (2024)
- [Variance reduction in output from generative AI](https://mlsanity.com/paper/2503.01033/) (2025)
- [The Dynamics of Collective Creativity in Human-AI Hybrid Societies](https://mlsanity.com/paper/2502.17962/) (2025)

## Built on by (most cited first, 1 of 1)

- [Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs](https://mlsanity.com/paper/2605.11128/) (2026)

## Abstract (arXiv)

The diversity of human expression is the raw material of discovery. Generative artificial intelligence threatens this resource even as it promises to accelerate innovation, a paradox now visible across science, culture, and professional work. We propose a framework to explain this tension. AI systems compress informational variance through statistical optimization, and users amplify this effect through epistemic deference. We call this process the AI Prism. Yet this same compression can enable novelty. Standardized forms travel across domain boundaries, lowering translation costs and creating opportunities for recombination that we term the Paradoxical Bridge. The interaction produces a U-shaped temporal dynamic, an initial decline in diversity followed by recombinant innovation, but only when humans actively curate rather than passively defer. The framework generates testable predictions about when compression constrains versus amplifies creativity. As AI becomes infrastructure for knowledge work, managing this dynamic is essential. Without intervention, the conditions for recovery may not arrive.