mlsanityData to 7 Oct 2026

Heterogeneous Agent Collaborative Reinforcement Learning

arXiv 2603.02604 ↗First posted 3 Mar 2026cs.LGcited by 13

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 54 of its 60 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

54 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025LLM Collaboration With Multi-Agent Reinforcement Learning6 Aug
  2. 2025MARFT: Multi-Agent Reinforcement Fine-Tuning21 Apr
  3. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning12 Mar
  4. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning25 Feb
  5. 2024Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning8 Oct

Other references · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  12. 2020Learning to summarize from human feedbackcited by 2,562
37 more
  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  4. 2020Knowledge Distillation: A Surveycited by 1,578
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  8. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  10. 2025Group Sequence Policy Optimizationcited by 788
  11. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  12. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  13. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  14. 2022Large Language Models Are Reasoning Teacherscited by 318
  15. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  16. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  17. 2018Large scale distributed neural network training through online distillationcited by 144
  18. 2025Geometric-Mean Policy Optimizationcited by 107
  19. 2023HIPTrack: Visual Tracking with Historical Promptscited by 71
  20. 2023Heterogeneous-Agent Reinforcement Learningcited by 67
  21. 2025DCPO: Dynamic Clipping Policy Optimizationcited by 38
  22. 2022SparseTT: Visual Tracking with Sparse Transformerscited by 36
  23. 2026Your Group-Relative Advantage Is Biasedcited by 32
  24. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  25. 2025SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Trackingcited by 23
  26. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
  27. 2025Rethinking Thinking Tokens: LLMs as Improvement Operatorscited by 20
  28. 2026Does Your Reasoning Model Implicitly Know When to Stop Thinking?cited by 17
  29. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
  30. 2026Automated Optimization Modeling via a Localizable Error-Driven Perspectivecited by 14
  31. 2026Real-Time Aligned Reward Model beyond Semanticscited by 12
  32. 2025Interpretable Unsupervised Joint Denoising and Enhancement for Real-World low-light Scenarioscited by 11
  33. 2025LD-RPS: Zero-Shot Unified Image Restoration via Latent Diffusion Recurrent Posterior Samplingcited by 9
  34. 2025RedOne: Revealing Domain-specific LLM Post-Training in Social Networking Servicescited by 6
  35. 2025SeeDNorm: Self-Rescaled Dynamic Normalizationcited by 6
  36. 2025Prompt-SID: Learning Structural Representation Prompt via Latent Diffusion for Single-Image Denoisingcited by 5
  37. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  2. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaborationcited by 10
  3. 2026UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systemscited by 6
  4. 2026Adaptive Robust Estimator for Multi-Agent Reinforcement Learningcited by 2
  5. 2026Policy Improvement Reinforcement Learningcited by 2
  6. 2026Uni-MDTrack: Learning Decoupled Memory and Dynamic States for Parameter-Efficient Visual Tracking in All Modalitycited by 1
  7. 2026ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capabilitycited by 1
  8. 2026Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMscited by 0
  9. 2026Constraint-First Reasoning: A Training-Free Protocol for Exploiting Answer-Space Constraints in Mathematical Problem Solvingcited by 0
  10. 2026WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Trainingcited by 0
  11. 2026When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipelinecited by 0
  12. 2026Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVRcited by 0
1 more
  1. 2026S3: Spectral Null-Space Swap Makes Reasoning Models Efficientcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Models
  2. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic
  3. 2026Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning
  4. 2025From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demos
  5. 2025Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization
  6. 2025End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning
  7. 2026Multi-Agent Reinforcement Learning via Agent-Specific Preference
  8. 2026Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning