mlsanityData to 13 Oct 2026

Building Multi-Task Agentic LLMs via Two-Phase Distillation

arXiv 2606.30044 ↗First posted 29 Jun 2026cs.LGcited by 1

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

A key step toward artificial general intelligence is to train models that can perform multiple tasks. In this paper, we study how to build such models by first training separate RL experts for individual tasks and then consolidating them via distillation, as an alternative to directly training a single model on mixed tasks. We show that off-policy distillation degrades in multi-task settings due to the mode-covering nature of forward KL: aggregating data from multiple tasks introduces a large number of behavioral modes that can exceed the student's capacity, forcing it to average across behaviors and leading to degraded performance. In contrast, on-policy distillation is mode-seeking but requires strong initialization. Inspired by these observations, we propose a two-phase approach: off-policy distillation followed by on-policy refinement. Evaluation across conversational agents and text-based games confirms that this two-phase approach matches single-task RL expert performance for each individual task, whereas off-policy or on-policy distillation alone fails to match this performance.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 27 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

23 of 27 references.

References · 23

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,384
  4. 2022Editing Models with Task Arithmeticcited by 1,101
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 873
  6. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 808
  7. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 786
  8. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 784
  9. 2021Merging Models with Fisher-Weighted Averagingcited by 617
  10. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 582
  11. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 240
  12. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 171
11 more
  1. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 147
  2. 2019Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?cited by 34
  3. 2025Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervisioncited by 24
  4. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 21
  5. 2024MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximationcited by 20
  6. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 20
  7. 2026From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agentscited by 17
  8. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 13
  9. 2025Dynamic Fisher-weighted Model Merging via Bayesian Optimizationcited by 12
  10. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Modelscited by 11
  11. 2026The Molecular Structure of Thought: Mapping the Topology of Long Chain-of-Thought Reasoningcited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Rethinking Self-Distillation for Multi-Teacher Capability Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
  2. 2024Sub-goal Distillation: A Method to Improve Small Language Agents
  3. 2026When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffs
  4. 2026Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents
  5. 2025AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes
  6. 2025ProST: Progressive Sub-task Training for Pareto-Optimal Multi-agent Systems Using Small Language Models
  7. 2026ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks
  8. 2026MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate