One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement learning (RL) across multiple domains can broaden the reasoning capabilities of large language models (LLMs), yet joint training often degrades individual-domain performance and can destabilize optimization. Existing work typically diagnoses such interference from a single-step view using first-order gradient alignment or curvature-based proxies. We show that this view can miss a critical form of sequential interference: same-point domain gradients may remain nearly orthogonal even when consecutive realized updates partially reverse one another in output space. We further show that consecutive token log-probability footprints recover this interaction directly from adjacent checkpoints as a local second-order interaction in output space, without explicitly reconstructing same-step curvature. Building on this insight, we propose OSOL, which designates a focus domain at each iteration, uses the preceding checkpoint footprint to rank token-level rebound risk, and applies a drift-ranked, adaptively scaled correction within the standard GRPO update. Our analysis shows that this correction suppresses the targeted cross-step output backtracking component. Controlled studies further show that cross-step backtracking is more strongly associated with subsequent task damage than same-point gradient diagnostics, while the preceding footprint ranks future rebound risk more accurately than Hessian-based proxies. On Qwen3-30B-A3B, OSOL reaches a domain-macro average of 0.4822, improving by 5.7% over the strongest compared baseline, without explicit higher-order differentiation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 37 references that are in the map. First posted on arXiv 6 Sep 2026.
Builds on
27 of 37 references.
Closest ancestors · 9
Same direction, newest first
- 2026A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL1 Jun
- 2026ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning1 May
- 2026Multi-Task GRPO: Reliable LLM Reasoning Across Tasks5 Feb
- 2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgery2 Feb
- 2026When Domains Interact: Asymmetric and Order-Sensitive Cross-Domain Effects in Reinforcement Learning for Reasoning1 Feb
- 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning23 Jul
- 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective17 Jun
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 18
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
- 2020Gradient Surgery for Multi-Task Learningcited by 1,256
- 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
- 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
- 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
- 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
- 2014New insights and perspectives on the natural gradient methodcited by 443
- 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
- 2022Multi-Task Learning as a Bargaining Gamecited by 215
- 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
6 more
- 2023FAMO: Fast Adaptive Multitask Optimizationcited by 78
- 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 20
- 2024CoBa: Convergence Balancer for Multitask Finetuning of Large Language Modelscited by 13
- 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modelingcited by 12
- 2025ResT: Reshaping Token-Level Policy Gradients for Tool-Use Large Language Modelscited by 10
- 2026Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVRcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Spectral Rewiring for Exploration, Purification, and Model Merging
- 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learning
- 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
- 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model
- 2026Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Critics
- 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correction
- 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
- 2026Joint Training of Multi-Token Prediction in Reinforcement Learning via Optimal Coefficient Calibration