mlsanityData to 14 Oct 2026

ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

arXiv 2605.00380 ↗First posted 1 May 2026cs.LG cs.CLcited by 6

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning of Large Language Models (LLMs) but usually exhibits limited generation diversity due to the over-incentivization of positive rewards. Although methods like Negative Sample Reinforcement (NSR) mitigate this issue by upweighting penalty from negative samples, they may suppress the semantic distributions shared between positive and negative responses. To boost reasoning ability without losing diversity, this paper proposes negative sample projection Residual Reinforcement Learning (ResRL) that decouples similar semantic distributions among positive and negative responses. We theoretically link Lazy Likelihood Displacement (LLD) to negative-positive head-gradient interference and derive a single-forward proxy that upper-bounds representation alignment to guide conservative advantage reweighting. ResRL then projects negative-token hidden representations onto an SVD-based low-rank positive subspace and uses projection residuals to modulate negative gradients, improving reasoning while preserving diversity and outperforming strong baselines on average across twelve benchmarks spanning Mathematics, Code, Agent Tasks, and Function Calling. Notably, ResRL surpasses NSR on mathematical reasoning by 9.4% in Avg@16 and 7.0% in Pass@128. Code is available at https://github.com/1229095296/ResRL.git.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 54 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

31 of 54 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacement3 Dec
  2. 2025Pass@k Metric for RLVR: A Diagnostic Tool of Exploration, But Not an Objective20 Nov
  3. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search29 Sep
  4. 2025FlowRL: Matching Reward Distributions for LLM Reasoning18 Sep
  5. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration19 Aug
  6. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  7. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR11 Aug
  8. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control5 Aug
  9. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  10. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  11. 2025Learning to Reason without External Rewards26 May
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
2 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 17

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2016Layer Normalizationcited by 5,669
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  10. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  11. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  12. 2024Token-level Direct Preference Optimizationcited by 135
5 more
  1. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  2. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 46
  3. 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
  4. 2025Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answerscited by 11
  5. 2025ResT: Reshaping Token-Level Policy Gradients for Tool-Use Large Language Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignmentcited by 2
  2. 2026Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoningcited by 1
  3. 2026Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learningcited by 1
  4. 2026When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agentscited by 1
  5. 2026One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Controlcited by 1
  6. 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Negative Reinforcement for LLM Reasoning:Dynamically Balancing Correction and Diversity in RLVR
  2. 2026Not All Negative Samples Are Equal: LLMs Learn Better from Plausible Reasoning
  3. 2025Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoning
  4. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning
  5. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  6. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  7. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  8. 2026From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space