mlsanityData to 7 Oct 2026

SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation

arXiv 2608.09271 ↗First posted 10 Aug 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group-based reinforcement learning objectives such as GRPO can allocate learning signal poorly across prompt difficulty: under binary rewards, group normalization induces a divergent weighting on easy prompts. We introduce Softmax Advantage Group Estimation (SoftmaxGRPO), a drop-in alternative that replaces z-score-normalized group advantages with temperature-scaled softmax advantages, keeping weights bounded regardless of prompt difficulty. For binary rewards, we derive the exact finite-group population objective and identify MaxRL as its low-temperature limit. For bounded scalar rewards, we show that the large-group update exactly optimizes a log-moment-generating-function objective, while a universal finite-group scalar objective cannot exist without additional assumptions on the reward distribution. Empirically, SoftmaxGRPO reallocates measured gradient budget away from near-solved prompts and consistently improves over GRPO under identical rewards. It reaches 51.8% on DeepMath with verifiable rewards and improves a 1.5B instruction-tuned model from 35.0% to 68.0% on Poetry using only lightweight text-similarity rewards.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 50 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

41 of 50 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Rethinking the Trust Region in LLM Reinforcement Learning4 Feb
  2. 2025Escaping the Verifier: Learning to Reason via Demonstrations26 Nov
  3. 2025What is the objective of reasoning with reinforcement learning?15 Oct
  4. 2025RLP: Reinforcement as a Pretraining Objective26 Sep
  5. 2025Reinforcing General Reasoning without Verifiers27 May
  6. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  7. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  8. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  12. 2015Policy Distillationcited by 414
17 more
  1. 2018Maximum a Posteriori Policy Optimisationcited by 318
  2. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  3. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  5. 2024Iterative Reasoning Preference Optimizationcited by 244
  6. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  7. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 122
  8. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 81
  9. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  10. 2025Learning to Reason for Long-Form Story Generationcited by 34
  11. 2026Maximum Likelihood Reinforcement Learningcited by 29
  12. 2025SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sportscited by 17
  13. 2017Cold-Start Reinforcement Learning with Softmax Policy Gradientcited by 15
  14. 2018Optimal Completion Distillation for Sequence Learningcited by 14
  15. 2025KAT-V1: Kwai-AutoThink Technical Reportcited by 12
  16. 2025ProxyThinker: Test-Time Guidance through Small Visual Reasonerscited by 8
  17. 2025Answer-Consistent Chain-of-thought Reinforcement Learning For Multi-modal Large Langauge Modelscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  2. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  3. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  4. 2026MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following
  5. 2026GVPO++: Group Variance Policy Optimization for LLM Post-Training and On-Policy Distillation
  6. 2026On the Hidden Objective Biases of Group-based Reinforcement Learning
  7. 2026Your Group-Relative Advantage Is Biased
  8. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation