mlsanityData to 7 Oct 2026

n-Musketeers: Reinforcement Learning Shapes Collaboration Among Language Models

arXiv 2602.09173 ↗First posted 9 Feb 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent progress in reinforcement learning with verifiable rewards (RLVR) shows that small, specialized language models (SLMs) can exhibit structured reasoning without relying on large monolithic LLMs. We introduce soft hidden-state collaboration, where multiple heterogeneous frozen SLM experts are integrated through their internal representations via a trainable attention interface. Experiments on Reasoning Gym and GSM8K show that this latent integration is competitive with strong single-model RLVR baselines. Ablations further reveal a dual mechanism of expert utilization: for simpler arithmetic domains, performance gains can largely be explained by static expert preferences, whereas more challenging settings induce increasingly concentrated and structured expert attention over training, indicating emergent specialization in how the router connects to relevant experts. Overall, hidden-state collaboration provides a compact mechanism for leveraging frozen experts, while offering an observational window into expert utilization patterns and their evolution under RLVR.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 53 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

44 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  9. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
27 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  4. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  6. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  7. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  8. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  9. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  10. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  11. 2025Small Language Models are the Future of Agentic AIcited by 254
  12. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  13. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  14. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  15. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  16. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66
  17. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 64
  18. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 61
  19. 2023Fusing Models with Complementary Expertisecited by 45
  20. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 28
  21. 2025Causal LLM Routing: End-to-End Regret Minimization from Observational Datacited by 27
  22. 2025Difficulty-Aware Agentic Orchestration for Query-Specific Multi-Agent Workflowscited by 23
  23. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  24. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  25. 2025R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expertscited by 12
  26. 2025Automated Composition of Agents: A Knapsack Approach for Agentic Component Selectioncited by 10
  27. 2025Brain-Like Processing Pathways Form in Models With Heterogeneous Expertscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
  2. 2025Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models
  3. 2026Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
  4. 2026Hide to Guide: Learning via Semantic Masking
  5. 2026Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
  6. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  7. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  8. 2026Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov States