mlsanityData to 7 Oct 2026

Spurious Rewards: Rethinking Training Signals in RLVR

arXiv 2506.10947 ↗First posted 12 Jun 2025cs.AI cs.LGcited by 199

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We show that reinforcement learning with verifiable rewards (RLVR) can elicit strong mathematical reasoning in certain language models even with spurious rewards that have little, no, or even negative correlation with the correct answer. For example, RLVR training with GRPO improves MATH-500 performance for Qwen2.5-Math-7B by 21.4 percentage points using randomly assigned rewards, nearly matching the 29.1-point gain from ground-truth rewards. To explain this counterintuitive observation, we show that GRPO exhibits a clipping bias from the clip term, which can amplify high-prior behaviors learned during pretraining even without informative rewards. As a case study, we identify one such behavior in Qwen2.5-Math models, which we call code reasoning -- reasoning in code without actual code execution; code-reasoning frequency increases from 65 percent to over 90 percent with spurious rewards. However, the presence of such amplifiable behaviors is highly model-dependent. In practice, spurious rewards that are effective for Qwen models often fail to produce gains for other model families, such as Llama3 or OLMo2. Our results highlight the importance of validating RL methods across diverse models rather than relying on a single de facto choice: large gains can arise on Qwen models even from random rewards that do not reflect genuine capability improvements.

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 52 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

38 of 52 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  2. 2025Maximizing Confidence Alone Improves Reasoning28 May
  3. 2025Can Large Reasoning Models Self-Train?27 May
  4. 2025Learning to Reason without External Rewards26 May
  5. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  6. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  7. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  9. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  10. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  11. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
8 more
  1. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  4. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  6. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct
  7. 2023Let's Verify Step by Step31 May
  8. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  6. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  7. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  8. 20242 OLMo 2 Furiouscited by 302
  9. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  10. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoningcited by 118
  11. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 92
  12. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
6 more
  1. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  2. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  3. 2025Concise Reasoning via Reinforcement Learningcited by 78
  4. 2024Self-Consistency Preference Optimizationcited by 50
  5. 2025Rethinking Reflection in Pre-Trainingcited by 43
  6. 2025One-shot Entropy Minimizationcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 84; 2026: 115.

199 papers.

Sort:

By area:

  1. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  2. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  3. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  4. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  6. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  7. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  8. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  9. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  10. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  11. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 84
  12. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 81
38 more
  1. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  2. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 64
  3. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  4. 2025ThetaEvolve: Test-time Learning on Open Problemscited by 52
  5. 2025Self-Questioning Language Modelscited by 47
  6. 2025First Return, Entropy-Eliciting Explorecited by 45
  7. 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 40
  8. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 39
  9. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
  10. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 32
  11. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  12. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 27
  13. 2025Cognitive Foundations for Reasoning and Their Manifestation in LLMscited by 26
  14. 2025Nudging the Boundaries of LLM Reasoningcited by 25
  15. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25
  16. 2026Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weightscited by 25
  17. 2025ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learningcited by 24
  18. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimizationcited by 23
  19. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 20
  20. 2025HeurAgenix: Leveraging LLMs for Solving Complex Combinatorial Optimization Challengescited by 19
  21. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 18
  22. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 17
  23. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gaincited by 16
  24. 2026Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMscited by 13
  25. 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 12
  26. 2025The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societiescited by 12
  27. 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Modelscited by 12
  28. 2026EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubricscited by 12
  29. 2025The Serial Scaling Hypothesiscited by 11
  30. 2025K2-Think: A Parameter-Efficient Reasoning Systemcited by 11
  31. 2026Learn Hard Problems During RL with Reference Guided Fine-tuningcited by 11
  32. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 11
  33. 2025ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Contextcited by 10
  34. 2025Reasoning Models are Test Exploiters: Rethinking Multiple-Choicecited by 10
  35. 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thoughtcited by 10
  36. 2025MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Modelcited by 10
  37. 2025OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoningcited by 10
  38. 2026Entropy-Preserving Reinforcement Learningcited by 10

50 most cited of 199 All 199 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  2. 2026When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study
  3. 2026Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
  4. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  5. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  6. 2026Reward Granularity in RLVR: Comparing Process and Outcome Reward Structures for Mathematical Reasoning in Small Language Models
  7. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers
  8. 2025The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason