mlsanityData to 14 Oct 2026

SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization

arXiv 2505.12346 ↗First posted 18 May 2025cs.AIcited by 89

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large language models (LLMs) exhibit varying levels of confidence across input prompts (questions): some lead to consistent, semantically similar answers, while others yield diverse or contradictory outputs. This variation reflects LLM's uncertainty about the input prompt, a signal of how confidently the model understands a given problem. However, vanilla Group Relative Policy Optimization (GRPO) treats all prompts equally during policy updates, ignoring this important information about the model's knowledge boundaries. To address this limitation, we propose SEED-GRPO (Semantic Entropy EnhanceD GRPO), which explicitly measures LLMs' uncertainty of the input prompts semantic entropy. Semantic entropy measures the diversity of meaning in multiple generated answers given a prompt and uses this to modulate the magnitude of policy updates. This uncertainty-aware training mechanism enables dynamic adjustment of policy update magnitudes based on question uncertainty. It allows more conservative updates on high-uncertainty questions while maintaining the original learning signal on confident ones. Experimental results on five mathematical reasoning benchmarks (AIME24 56.7, AMC 68.7, MATH 83.4, Minerva 34.2, and OlympiadBench 48.0) demonstrate that SEED-GRPO achieves new state-of-the-art performance in average accuracy, validating the effectiveness of uncertainty-aware policy optimization.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 42 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

29 of 42 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  2. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  3. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  4. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  5. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  6. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  7. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  8. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  9. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  11. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
3 more
  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024OpenAI o1 System Cardcited by 1,244
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  11. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  12. 2025LIMO: Less is More for Reasoningcited by 498
2 more
  1. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  2. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 36; 2026: 53.

89 papers.

Sort:

By area:

  1. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  2. 2025Learning to Reason without External Rewardscited by 210
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2026Self-Distilled RLVRcited by 131
  5. 2025Geometric-Mean Policy Optimizationcited by 111
  6. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  7. 2025ExGRPO: Learning to Reason from Experiencecited by 53
  8. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 46
  9. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learningcited by 42
  10. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversitycited by 41
  11. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  12. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
38 more
  1. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  2. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 31
  3. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learningcited by 21
  4. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  5. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoningcited by 19
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  7. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Frameworkcited by 14
  8. 2025BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuningcited by 14
  9. 2026What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agentscited by 13
  10. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 12
  11. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  12. 2025Scheduling Your LLM Reinforcement Learning with Reasoning Treescited by 12
  13. 2026Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMscited by 12
  14. 2025MAPO: Mixed Advantage Policy Optimizationcited by 11
  15. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 11
  16. 2025Adaptive Termination for Multi-round Parallel Reasoning: An Universal Semantic Entropy-Guided Frameworkcited by 10
  17. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  18. 2026Reinforcement Learning from Rich Feedback with Distributional DAggercited by 10
  19. 2026STRIDE: Strategic Trajectory Reasoning via Discriminative Estimation for Verifiable Reinforcement Learningcited by 10
  20. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 9
  21. 2026Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoningcited by 9
  22. 2026Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thoughtcited by 9
  23. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  24. 2026Self-Distilled Policy Gradientcited by 8
  25. 2026TeleBoost: A Systematic Alignment Framework for High-Fidelity, Controllable, and Robust Video Generationcited by 7
  26. 2026Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimizationcited by 7
  27. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweightingcited by 6
  28. 2026Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantificationcited by 6
  29. 2025From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Trainingcited by 5
  30. 2026AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Modelscited by 5
  31. 2026EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Modelscited by 5
  32. 2026CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVRcited by 5
  33. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  34. 2025Thinking with Images via Self-Calling Agentcited by 4
  35. 2026PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewardscited by 4
  36. 2026DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replaycited by 4
  37. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  38. 2026Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifierscited by 4

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
  2. 2026Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
  3. 2026GRPO Training Dynamics for Small Language Models
  4. 2026Taming Extreme Tokens: Covariance-Aware GRPO with Gaussian-Kernel Advantage Reweighting
  5. 2026N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization
  6. 2026UCPO: Uncertainty-Aware Policy Optimization
  7. 2026The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
  8. 2025Information-Consistent Language Model Recommendations through Group Relative Policy Optimization