mlsanityData to 7 Oct 2026

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

arXiv 2608.03573 ↗First posted 4 Aug 2026cs.CL cs.LGcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 65 of its 77 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

65 of 77 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space15 Apr
  2. 2025The Path Not Taken: RLVR Provably Learns Off the Principals11 Nov
  3. 2025On Predictability of Reinforcement Learning Dynamics for Large Language Models1 Oct
  4. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs25 Sep
  5. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning23 Jul
  6. 2025Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling20 Jul
  7. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  8. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective17 Jun
  9. 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond26 May
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
2 more
  1. 2023Let's Verify Step by Step31 May
  2. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 51

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
38 more
  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2025Group Sequence Policy Optimizationcited by 788
  4. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  6. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  7. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  8. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  9. 2025Kimi K2: Open Agentic Intelligencecited by 523
  10. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  11. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  12. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  13. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  14. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  15. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  16. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
  17. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  18. 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
  19. 2025When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMscited by 51
  20. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  21. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 43
  22. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42
  23. 2024Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuningcited by 27
  24. 2025Not All Correct Answers Are Equal: Why Your Distillation Source Matterscited by 25
  25. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videoscited by 22
  26. 2025Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policiescited by 22
  27. 2025Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agentscited by 19
  28. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 19
  29. 2025DeepDistill: Enhancing LLM Reasoning Capabilities via Large-Scale Difficulty-Graded Data Trainingcited by 17
  30. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  31. 2024Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Modelscited by 15
  32. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  33. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoningcited by 8
  34. 2023DeepScaler: Holistic Autoscaling for Microservices Based on Spatiotemporal GNN with Adaptive Graph Learningcited by 7
  35. 2026Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix Itcited by 6
  36. 2025Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modelingcited by 4
  37. 2026Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planningcited by 4
  38. 2026Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Docited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Enhancing Diffusion Language Models with Autoregressive Post-Training Weightscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
  2. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMs
  3. 2026Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
  4. 2025Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
  5. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
  6. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
  7. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy
  8. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions