mlsanityData to 14 Oct 2026

Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting

arXiv 2508.05928 ↗First posted 8 Aug 2025cs.LGcited by 6

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group-Relative Policy Optimization (GRPO) is a key technique for training large reasoning models, yet it suffers from a critical vulnerability: the Think-Answer Mismatch, where noisy reward signals corrupt the learning process. This problem is most severe in unbalanced response groups, paradoxically degrading the signal precisely when it should be most informative. To address this challenge, we propose Stable Group-Relative Policy Optimization (S-GRPO), a principled enhancement that derives optimal, noise-aware advantage weights to stabilize training. Our comprehensive experiments on mathematical reasoning benchmarks demonstrate S-GRPO's effectiveness and robustness. On various models, S-GRPO significantly outperforms DR. GRPO, achieving performance gains of +2.5% on Qwen-Math-7B-Base, +2.2% on Llama-3.2-3B-Base, and +2.4% on Qwen-Math-1.5B-Instruct. Most critically, while standard GRPO fails to learn under 20% synthetic reward noise, S-GRPO maintains stable learning progress. These results highlight S-GRPO's potential for more robust and effective training of large-scale reasoning models. footnoteCode and data are available at: https://github.com/shenpeijun0212/S-GRPO

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 35 references that are in the map. First posted on arXiv 8 Aug 2025.

Builds on

28 of 35 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  3. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  4. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  5. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  6. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  7. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  8. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't20 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
5 more
  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan
  3. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoning9 Dec
  4. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2025s1: Simple test-time scalingcited by 1,341
  4. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
  5. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  6. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
  7. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  8. 2025Are Reasoning Models More Prone to Hallucination?cited by 64
  9. 2024Bayesian Reward Models for LLM Alignmentcited by 35
  10. 2025Long Is More Important Than Difficult for Training Reasoning Modelscited by 11
  11. 2025On Symmetric Losses for Robust Policy Optimization with Noisy Preferencescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Your LLM Agents are Temporally Blind: The Misalignment Between Tool Use Decisions and Human Time Perceptioncited by 9
  2. 2025VIDEOP2R: Video Understanding from Perception to Reasoningcited by 8
  3. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoningcited by 4
  4. 2025PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMscited by 4
  5. 2026Consensus Group Relative Policy Optimization for Text Generationcited by 1
  6. 2026DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Are complicated loss functions necessary for teaching LLMs to reason?
  2. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
  3. 2026Calibration-Aware Policy Optimization for Reasoning LLMs
  4. 2026Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
  5. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  6. 2026GRPO Training Dynamics for Small Language Models
  7. 2025Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning
  8. 2025DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization