mlsanityData to 7 Oct 2026

The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning

arXiv 2505.15134 ↗First posted 21 May 2025cs.LG cs.AIcited by 160

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Entropy minimization (EM) trains the model to concentrate even more probability mass on its most confident outputs. We show that this simple objective alone, without any labeled data, can substantially improve large language models' (LLMs) performance on challenging math, physics, and coding tasks. We explore three approaches: (1) EM-FT minimizes token-level entropy similarly to instruction finetuning, but on unlabeled outputs drawn from the model; (2) EM-RL: reinforcement learning with negative entropy as the only reward to maximize; (3) EM-INF: inference-time logit adjustment to reduce entropy without any training data or parameter updates. On Qwen-7B, EM-RL, without any labeled data, achieves comparable or better performance than strong RL baselines such as GRPO and RLOO that are trained on 60K labeled examples. Furthermore, EM-INF enables Qwen-32B to match or exceed the performance of proprietary models like GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro on the challenging SciCode benchmark, while being 3x more efficient than self-consistency and sequential refinement. Our findings reveal that many pretrained LLMs possess previously underappreciated reasoning capabilities that can be effectively elicited through entropy minimization alone, without any labeled data or even any parameter updates.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 77 of its 96 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

77 of 96 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  2. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  3. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  4. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  5. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  9. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 67

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2020Learning to summarize from human feedbackcited by 2,562
38 more
  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2025s1: Simple test-time scalingcited by 1,311
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  13. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  14. 2022Large Language Models Can Self-Improvecited by 656
  15. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  16. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  17. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  18. 2019Semi-supervised Domain Adaptation via Minimax Entropycited by 333
  19. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  20. 2019A Baseline for Few-Shot Image Classificationcited by 298
  21. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  22. 2022Parameter-free Online Test-time Adaptationcited by 184
  23. 2021InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Renderingcited by 180
  24. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  25. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  26. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  27. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  28. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  29. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 137
  30. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  31. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  32. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  33. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  34. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  35. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  36. 2019Unsupervised Domain Adaptation using Feature-Whitening and Consensus Losscited by 67
  37. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
  38. 2023Universal Test-time Adaptation through Weight Ensembling, Diversity Weighting, and Prior Correctioncited by 65

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 72; 2026: 88.

160 papers.

Sort:

By area:

  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  2. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  3. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  4. 2025Learning to Reason without External Rewardscited by 199
  5. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  6. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  7. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 83
  8. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 81
  9. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 48
  10. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  11. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  12. 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 40
38 more
  1. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 38
  2. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  3. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  4. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 36
  5. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
  6. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  7. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 33
  8. 2025One-shot Entropy Minimizationcited by 32
  9. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 32
  10. 2025Self-Improving LLM Agents at Test-Timecited by 30
  11. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 26
  12. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
  13. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25
  14. 2025RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agentscited by 24
  15. 2025Language Models that Think, Chat Bettercited by 23
  16. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimizationcited by 23
  17. 2025Interleaved Reasoning for Large Language Models via Reinforcement Learningcited by 21
  18. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learningcited by 21
  19. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 20
  20. 2025Revisiting LLM Reasoning via Information Bottleneckcited by 19
  21. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 19
  22. 2025Semantic Energy: Detecting LLM Hallucination Beyond Entropycited by 18
  23. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 17
  24. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 15
  25. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalizationcited by 13
  26. 2026Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillationcited by 13
  27. 2025Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learningcited by 12
  28. 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Modelscited by 12
  29. 2025Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMscited by 12
  30. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selectioncited by 11
  31. 2025Unsupervised Elicitation of Language Modelscited by 11
  32. 2025K2-Think: A Parameter-Efficient Reasoning Systemcited by 11
  33. 2025Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Naturecited by 11
  34. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 11
  35. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoningcited by 11
  36. 2025SuperFlow: Training Flow Matching Models with RL on the Flycited by 10
  37. 2026Reward-free Alignment for Conflicting Objectivescited by 10
  38. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rarecited by 10

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
  2. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  3. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
  4. 2025Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoning
  5. 2026Think Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning
  6. 2025Maximizing Confidence Alone Improves Reasoning
  7. 2026Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning
  8. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning