mlsanityData to 14 Oct 2026

A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL

arXiv 2606.02398 ↗First posted 1 Jun 2026cs.LG cs.CLcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) post-training improves large language models (LLMs) on individual domains such as mathematical reasoning, code generation, question answering, and creative writing (CW), but training on one domain often degrades performance on others. Existing explanations based on catastrophic forgetting or global gradient conflict are incomplete: substantial interference can occur even when full-model gradients are nearly orthogonal. We show that single-domain RL produces sparse, small-magnitude parameter edits with weak overlap among top-changed neurons, while different domains still share substantial active computation routes on which update directions determine whether they act synergistically or conflict. Guided by this observation, we prove under a local perturbation model of multi-domain RL that later-domain training harms an earlier domain mainly through a second-order damage term, which under the observed sparse route structure concentrates in a low-dimensional shared conflict subspace. Moreover, a short domain refresh contracts the harmful component on this subspace, enabling selective recovery with limited collateral damage. Consistent with the theory, a brief Re-Math refresh after Code rightarrow Math rightarrow QA rightarrow CW recovers Math from 57.66 to 66.04 while largely preserving performance on the other domains, yielding the best average score of 66.39. Beyond refresh, a training-free rollback on a sparse proxy conflict coordinate set for the Math-QA pair partially restores Math, providing direct proxy-level evidence for localized damage. These results provide a localized mechanistic account of interference and recovery in multi-domain RL.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 40 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

33 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models27 Apr
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2023Let's Verify Step by Step31 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 29

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
17 more
  1. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  3. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  4. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  5. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  6. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
  7. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  8. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 176
  9. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  10. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  11. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 106
  12. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  13. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 94
  14. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 66
  15. 2025Spurious Forgetting in Continual Learning of Language Modelscited by 51
  16. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 47
  17. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Controlcited by 1
  2. 2026Beyond Static Interpretability: Anticipating Post-SFT Mechanisms from Pre-SFT Parameters for Better Tuningcited by 0
  3. 2026A helps B while B hurts A: directed transfer in instruction-tuning mixturecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Spectral Rewiring for Exploration, Purification, and Model Merging
  2. 2025Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models
  3. 2026When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoning
  4. 2026SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
  5. 2026Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
  6. 2025Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?
  7. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
  8. 2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff