mlsanityData to 14 Oct 2026

Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization

arXiv 2504.05812 ↗First posted 8 Apr 2025cs.LGcited by 128

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Existing methods to enhance the reasoning capability of large language models predominantly rely on supervised fine-tuning (SFT) followed by reinforcement learning (RL) on reasoning-specific data. These approaches critically depend on external supervisions--such as labeled reasoning traces, verified golden answers, or pre-trained reward models. In this work, we propose Entropy Minimized Policy Optimization (ours), which makes an early attempt at fully unsupervised LLM reasoning incentivization. By continuously minimizing the predictive entropy of LLMs on unlabeled questions in a latent semantic space, ours achieves competitive performance compared to supervised counterparts on both mathematical and free-form natural reasoning tasks. Specifically, without any supervised signals, ours boosts the accuracy of Qwen2.5-Math-7B Base from 30.7% to 48.1% on mathematical benchmarks and improves the accuracy of Qwen2.5-7B Base from 32.1% to 50.1% on MMLU-Pro. Primary experiments and analysis are also provided to interpret the effectiveness of ours. Code is available at https://github.com/QingyangZhang/EMPO.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 55 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

29 of 55 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025Process Reinforcement through Implicit Rewards3 Feb
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2024Free Process Rewards without Process Labels2 Dec
  6. 2024CREAM: Consistency Regularized Self-Rewarding Language Models16 Oct
  7. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 21

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  10. 2022Large Language Models Can Self-Improvecited by 664
  11. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  12. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
9 more
  1. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  2. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  3. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 84
  4. 2025Self-rewarding correction for mathematical reasoningcited by 70
  5. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 55
  6. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 39
  7. 2024COME: Test-time adaption by Conservatively Minimizing Entropycited by 31
  8. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  9. 2024The Entropy Enigma: Success and Failure of Entropy Minimizationcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 62; 2026: 66.

128 papers.

Sort:

By area:

  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  2. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  4. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  5. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  6. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  7. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  8. 2025Learning to Reason without External Rewardscited by 210
  9. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  10. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  11. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
  12. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
38 more
  1. 2025Geometric-Mean Policy Optimizationcited by 111
  2. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 89
  3. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 51
  4. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 50
  5. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 46
  6. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 46
  7. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  8. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversitycited by 41
  9. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 41
  10. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 37
  11. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  12. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  13. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  14. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 31
  15. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 31
  16. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 29
  17. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 27
  18. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 27
  19. 2025RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agentscited by 25
  20. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 23
  21. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  22. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoningcited by 19
  23. 2025Semantic Energy: Detecting LLM Hallucination Beyond Entropycited by 18
  24. 2026On-Policy Self-Distillation without Any Supervisioncited by 17
  25. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalizationcited by 15
  26. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  27. 2025SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translationcited by 13
  28. 2025Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learningcited by 12
  29. 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Timecited by 12
  30. 2025Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answerscited by 11
  31. 2026Entropy-Preserving Reinforcement Learningcited by 11
  32. 2026Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvementcited by 11
  33. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  34. 2025Self-Evolving Vision-Language Models for Image Quality Assessment via Voting and Rankingcited by 10
  35. 2026VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reductioncited by 10
  36. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  37. 2025ESPO: Entropy Importance Sampling Policy Optimizationcited by 9
  38. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9

50 most cited of 128 All 128 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
  2. 2025Maximizing Confidence Alone Improves Reasoning
  3. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains
  4. 2026Unified Data Selection for LLM Reasoning
  5. 2026Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning
  6. 2026Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoning
  7. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
  8. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning