mlsanityData to 14 Oct 2026

From Collaboration to Capability: Internalizing Routed LLM Experts into Compact Reasoners

arXiv 2609.12578 ↗First posted 11 Sep 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

A compact controller can coordinate stronger experts by selecting whom to consult, formulating requests, and integrating their responses. We study whether learning from both the controller's decisions and the experts' reasoning and code improves its generation after expert removal. We introduce textscRivet for collaboration internalization: expert-augmented reinforcement learning applies a shared outcome signal to controller decisions and returned expert spans, and verified trajectory internalization consolidates complete successful interactions through format-aware supervised training. The deployed controller generates reasoning, code, and interaction structure with local Python execution and no external LLM. Across seven competition-mathematics benchmarks, RIVET-1.7B and RIVET-4B achieve average accuracies of 28.25% and 44.16%; Stage~II improves RIVET-4B's accuracy after expert removal by 6.49 points, and GPQA-Diamond results provide evidence of generalization to scientific reasoning. Ablations show gains from ordinary trajectory supervision and additional format weighting, supporting the effectiveness of training on the content and structure of verified collaborations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 36 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

30 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Agentic Reinforced Policy Optimization26 Jul

Other references · 29

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  5. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  6. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  7. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  8. 2025ToRL: Scaling Tool-Integrated RLcited by 169
  9. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  10. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  11. 2025Towards Robust Mathematical Reasoningcited by 73
  12. 2025Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learningcited by 72
17 more
  1. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 72
  2. 2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 47
  3. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 47
  4. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  5. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 34
  6. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  7. 2026OPID: On-Policy Skill Distillation for Agentic Reinforcement Learningcited by 32
  8. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  9. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  10. 2026AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agentcited by 13
  11. 2026ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Taskscited by 11
  12. 2026Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMscited by 9
  13. 2026Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix Itcited by 7
  14. 2026Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learningcited by 6
  15. 2026Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learningcited by 4
  16. 2026PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agentscited by 3
  17. 2026Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026n-Musketeers: Reinforcement Learning Shapes Collaboration Among Language Models
  2. 2026Requesting Expert Reasoning: Augmenting LLM Agents with Learned Collaborative Intervention
  3. 2025Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improve
  4. 2026CollabFlow: Recursive Self-Improvement of Agent Collaboration
  5. 2025GroundAct: Can LLM Agents Ground Actions in Environmental States?
  6. 2025Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?
  7. 2025Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
  8. 2026Learning What to Retain: Gated-Memory Routing for Efficient Collaboration in Multi-Agent LLM Systems