# Entropy Centroids as Intrinsic Rewards for Test-Time Scaling arXiv: https://arxiv.org/abs/2604.26173 · HTML: https://mlsanity.com/paper/2604.26173/ ## Key facts - No citing papers in the map yet (data through 14 Oct 2026). - Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. - Builds on 40 of its 57 references that are in the map. - First posted on arXiv 28 Apr 2026. ## Placed in - LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/) - LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/) - Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/) ## Builds on (most cited first, 30 of 40) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022) - [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019) - [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024) - [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025) - [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024) - [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023) - [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025) - [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025) - [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024) - [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025) - [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://mlsanity.com/paper/2309.03883/) (2023) - [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024) - [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025) - [MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention](https://mlsanity.com/paper/2506.13585/) (2025) - [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025) - [Deep Think with Confidence](https://mlsanity.com/paper/2508.15260/) (2025) - [Scalable Best-of-N Selection for Large Language Models via Self-Certainty](https://mlsanity.com/paper/2502.18581/) (2025) - [Olmo 3](https://mlsanity.com/paper/2512.13961/) (2025) - [Language Model Cascades: Token-level uncertainty and beyond](https://mlsanity.com/paper/2404.10136/) (2024) - [Ministral 3](https://mlsanity.com/paper/2601.08584/) (2026) - [Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models](https://mlsanity.com/paper/2506.04210/) (2025) - [SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond](https://mlsanity.com/paper/2505.19641/) (2025) - [In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation](https://mlsanity.com/paper/2403.01548/) (2024) - [Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation](https://mlsanity.com/paper/2509.15194/) (2025) - [Towards Autonomous Mathematics Research](https://mlsanity.com/paper/2602.10177/) (2026) - [MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation](https://mlsanity.com/paper/2502.12468/) (2025) - [Diverse Preference Learning for Capabilities and Alignment](https://mlsanity.com/paper/2511.08594/) (2025) - [Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoning](https://mlsanity.com/paper/2510.08146/) (2025) - [MARS-SQL: A multi-agent reinforcement learning framework for Text-to-SQL](https://mlsanity.com/paper/2511.01008/) (2025) ## Abstract (arXiv) An effective way to scale up test-time compute of large language models is to sample multiple responses and then select the best one, as in Grok Heavy and Gemini Deep Think. Existing selection methods often rely on external reward models, which requires training a strong reward model and introduces additional computation overhead. As an alternative, previous approaches have explored intrinsic signals, such as confidence and entropy, but these signals are noisy with naive aggregation. In this work, we observe that high-entropy tokens tend to cluster into consecutive groups during inference, providing a more stable notion of model uncertainty than individual tokens. Together, these clusters reveal temporal patterns of model uncertainty throughout the inference process. Motivated by this observation, we propose to use the temporal structure of uncertainty as an intrinsic reward. To this end, we first formalize the basic unit of segment-level uncertainty as the High Entropy Phase (HEP), a variable-length segment that begins at a high-entropy token and ends when consecutive low-entropy tokens appear. We then define the Entropy Centroid, inspired by the concept of the center of mass in physics, as the weighted average position of all HEPs along the trajectory. Intuitively, a lower centroid indicates early exploration followed by confident generation, which we find often corresponds to higher response quality. Based on this insight, we propose the Lowest Centroid method, which selects the response with the lowest entropy centroid among multiple candidates. Experiments on mathematics, code generation, logical reasoning, and agentic tasks, across model scales ranging from 14B to 480B, show that Lowest Centroid consistently outperforms existing baselines and delivers stable gains as model size increases. Code is available at https://github.com/hkust-nlp/entropy-centroid.