# LLM Probability Concentration: How Alignment Shrinks the Generative Horizon

arXiv: https://arxiv.org/abs/2506.17871 · HTML: https://mlsanity.com/paper/2506.17871/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 70 of its 95 references that are in the map.
- First posted on arXiv 22 Jun 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 70)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training](https://mlsanity.com/paper/2309.17179/) (2023)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [RLHF Workflow: From Reward Modeling to Online RLHF](https://mlsanity.com/paper/2405.07863/) (2024)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)

## Built on by (most cited first, 26 of 26)

- [KL-Regularized Reinforcement Learning is Designed to Mode Collapse](https://mlsanity.com/paper/2510.20817/) (2025)
- [Entropy After </Think> for reasoning model early exiting](https://mlsanity.com/paper/2509.26522/) (2025)
- [Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards](https://mlsanity.com/paper/2509.24981/) (2025)
- [Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning](https://mlsanity.com/paper/2510.05251/) (2025)
- [Don't Throw Away Your Pretrained Model](https://mlsanity.com/paper/2510.09913/) (2025)
- [Optimizing Diversity and Quality through Base-Aligned Model Collaboration](https://mlsanity.com/paper/2511.05650/) (2025)
- [The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Models](https://mlsanity.com/paper/2604.24698/) (2026)
- [Mixture-of-Minds: Multi-Agent Reinforcement Learning for Table Understanding](https://mlsanity.com/paper/2510.20176/) (2025)
- [Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models](https://mlsanity.com/paper/2603.24844/) (2026)
- [EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts](https://mlsanity.com/paper/2606.18967/) (2026)
- [One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL](https://mlsanity.com/paper/2608.12253/) (2026)
- [The One-Word Census: Answer-Choice Conformity Across 105 Language Models](https://mlsanity.com/paper/2607.12796/) (2026)
- [The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety](https://mlsanity.com/paper/2601.06116/) (2026)
- [PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching](https://mlsanity.com/paper/2603.18363/) (2026)
- [Process Supervision of Confidence Margin for Calibrated LLM Reasoning](https://mlsanity.com/paper/2604.23333/) (2026)
- [Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs](https://mlsanity.com/paper/2605.11128/) (2026)
- [Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning](https://mlsanity.com/paper/2606.03962/) (2026)
- [Communication Enhances LLMs' Stability in Strategic Thinking](https://mlsanity.com/paper/2602.06081/) (2026)
- [DMAP: A Distribution Map for Text](https://mlsanity.com/paper/2602.11871/) (2026)
- [Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models](https://mlsanity.com/paper/2604.02340/) (2026)
- [Can You Keep a Secret? Involuntary Information Leakage in Language Model Writing](https://mlsanity.com/paper/2605.10794/) (2026)
- [How Language Models Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures](https://mlsanity.com/paper/2606.06635/) (2026)
- [Towards Physical Intuitions for Alignment Dynamics: A Case Study With Randomness Crystallization](https://mlsanity.com/paper/2606.29933/) (2026)
- [Iterative Finetuning is Mostly Idempotent](https://mlsanity.com/paper/2605.01130/) (2026)
- [Fine-Tuning Improves Information Conveyance in Language Models](https://mlsanity.com/paper/2605.30844/) (2026)
- [Rethinking Probability-Based Reinforcement Learning From Posterior Concentration](https://mlsanity.com/paper/2610.01458/) (2026)

## Abstract (arXiv)

Despite their impressive capabilities, aligned large language models (LLMs) often generate outputs that lack diversity. What drives this consistency in the generation? We investigate this phenomenon through the lens of probability concentration in the model's output distribution. To quantify it, we use the Branching Factor (BF)--the exponentiated length-averaged entropy of the output distribution, interpreted as the effective number of plausible next steps during generation. Our empirical analysis reveals two key findings: (1) BF often decreases as generation progresses, suggesting that LLMs become more predictable; a controlled intervention indicates that this decline is largely a task-independent property of autoregressive self-conditioning, distinct from alignment, and can be locally reversed by unexpected context. (2) Alignment tuning sharpens the output distribution from the outset, reducing BF by a factor of 2--5 overall and up to an order of magnitude (e.g., from 12 to 1.2) at early positions. This reduction helps explain why aligned models are less sensitive to decoding strategies. It also has implications for complex reasoning: aligned Chain-of-Thought (CoT) models (e.g., DeepSeek-distilled models) generate longer reasoning chains that reach later, more deterministic (lower-BF) stages, yielding more stable outputs. We hypothesize that alignment does not fundamentally change model behavior, but instead steers the model toward stylistic tokens (e.g., ``Sure'') that unlock low-entropy trajectories already present in the base model. Nudging experiments support this view: prompting base models with such tokens similarly reduces BF. Together, our findings establish BF as a diagnostic for understanding and controlling LLM outputs, clarifying how alignment reduces variability, CoT stabilizes generation, and base models can be steered away from diversity.