mlsanityData to 7 Oct 2026

Advancing General-Purpose Reasoning Models with Modular Gradient Surgery

arXiv 2602.02301 ↗First posted 2 Feb 2026cs.CL cs.AI cs.LGcited by 9

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has played a central role in recent advances in large reasoning models (LRMs), yielding strong gains in verifiable and open-ended reasoning. However, training a single general-purpose LRM across diverse domains remains challenging due to pronounced domain heterogeneity. Through a systematic study of two widely used strategies, Sequential RL and Mixed RL, we find that both incur substantial cross-domain interference at the behavioral and gradient levels, resulting in limited overall gains. To address these challenges, we introduce **M**odular **G**radient **S**urgery (**MGS**), which resolves gradient conflicts at the module level within the transformer. When applied to Llama and Qwen models, MGS achieves average improvements of 4.3 (16.6%) and 4.5 (11.1%) points, respectively, over standard multi-task RL across three representative domains (math, general chat, and instruction following). Further analysis demonstrates that MGS remains effective under prolonged training. Overall, our study clarifies the sources of interference in multi-domain RL and presents an effective solution for training general-purpose LRMs.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 76 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

52 of 76 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models15 Dec
  2. 2025Zero Reinforcement Learning Towards General Domains29 Oct
  3. 2025Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models24 Oct
  4. 2025Language Models that Think, Chat Better24 Sep
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  6. 2025ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search27 Aug
  7. 2025Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling20 Jul
  8. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  9. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  10. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  11. 2025Reward Reasoning Model20 May
  12. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
4 more
  1. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2024OpenAI o1 System Cardcited by 1,220
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  9. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  10. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  11. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  12. 2023Function Vectors in Large Language Modelscited by 292
24 more
  1. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  2. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  3. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  4. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  5. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
  6. 2025Generalizing Verifiable Instruction Followingcited by 178
  7. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
  8. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  9. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  10. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  11. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  12. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 82
  13. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  14. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
  15. 2025Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-rankingcited by 30
  16. 2025Deep Research: A Systematic Surveycited by 23
  17. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 19
  18. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  19. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15
  20. 2025Xiaomi MiMo-VL-Miloco Technical Reportcited by 15
  21. 2025How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidencecited by 12
  22. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  23. 2025TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inferencecited by 8
  24. 2025Soft Conflict-Resolution Decision Transformer for Offline Multi-Task Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal Statescited by 2
  2. 2026Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Modelscited by 2
  3. 2026Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMscited by 2
  4. 2026Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedbackcited by 1
  5. 2026Think Big, Search Small: Where Capacity Matters in Hierarchical Search Agents?cited by 1
  6. 2026One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Controlcited by 1
  7. 2026GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimizationcited by 0
  8. 2026CARE-RL: Capability-Aware Reinforcement Learning for Mitigating Cross-Domain Conflictscited by 0
  9. 2026Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  2. 2026Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient Reasoning
  3. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning
  4. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  5. 2026DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning
  6. 2025Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic
  7. 2025Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?
  8. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models