# Decoding in Geometry: Alleviating Embedding-Space Crowding for Complex Reasoning

arXiv: https://arxiv.org/abs/2601.22536 · HTML: https://mlsanity.com/paper/2601.22536/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs](https://mlsanity.com/paper/2407.01082/) (2024)
- [Truncation Sampling as Language Model Desmoothing](https://mlsanity.com/paper/2210.15191/) (2022)
- [A Thorough Examination of Decoding Methods in the Era of LLMs](https://mlsanity.com/paper/2402.06925/) (2024)
- [Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation](https://mlsanity.com/paper/2305.09860/) (2023)
- [Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding](https://mlsanity.com/paper/2405.02750/) (2024)
- [EDT: Improving Large Language Models' Generation by Entropy-based Dynamic Temperature Sampling](https://mlsanity.com/paper/2403.14541/) (2024)
- [RePO: Replay-Enhanced Policy Optimization](https://mlsanity.com/paper/2506.09340/) (2025)
- [Shared Global and Local Geometry of Language Model Embeddings](https://mlsanity.com/paper/2503.21073/) (2025)
- [Non-myopic Generation of Language Models for Reasoning and Planning](https://mlsanity.com/paper/2410.17195/) (2024)
- [Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations](https://mlsanity.com/paper/2408.15417/) (2024)
- [Solving a Million-Step LLM Task with Zero Errors](https://mlsanity.com/paper/2511.09030/) (2025)
- [Token embeddings violate the manifold hypothesis](https://mlsanity.com/paper/2504.01002/) (2025)
- [Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputs](https://mlsanity.com/paper/2510.01218/) (2025)
- [φ-Decoding: Adaptive Foresight Sampling for Balanced Inference-Time Exploration and Exploitation](https://mlsanity.com/paper/2503.13288/) (2025)
- [The structure of the token space for large language models](https://mlsanity.com/paper/2410.08993/) (2024)
- [Semantic-guided Diverse Decoding for Large Language Model](https://mlsanity.com/paper/2506.23601/) (2025)
- [Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexity](https://mlsanity.com/paper/2007.14966/) (2020)
- [Controlled LLM Decoding via Discrete Auto-regressive Biasing](https://mlsanity.com/paper/2502.03685/) (2025)

## Built on by (most cited first, 1 of 1)

- [Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning](https://mlsanity.com/paper/2609.18723/) (2026)

## Abstract (arXiv)

Sampling-based decoding underlies complex reasoning in large language models (LLMs), where decoding strategies critically shape model behavior. Temperature- and truncation-based methods reshape the next-token distribution through global probability reweighting or thresholding to balance the quality-diversity tradeoff. However, they operate solely on token probabilities, ignoring fine-grained relationships among tokens in the embedding space. We uncover a novel phenomenon, embedding-space crowding, where the next-token distribution concentrates its probability mass on geometrically close tokens in the embedding space. We quantify crowding at multiple granularities and find a statistical association with reasoning success in mathematical problem solving. Motivated by this finding, we propose CraEG, a plug-and-play sampling method that mitigates crowding through geometry-guided reweighting. CraEG is training-free, single-pass, and compatible with standard sampling strategies. Experiments on multiple models and benchmarks demonstrate improved generation performance, with gains in robustness and diversity metrics.