mlsanityData to 14 Oct 2026

Improving LLM Collaboration via Multi-Agent Preference Learning

arXiv 2609.32827 ↗First posted 26 Sep 2026cs.AI cs.MAcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents also fits: General LLM agent frameworks and benchmarks

also in 1 more
Abstract (arXiv)

Several works have explored multi-agent reinforcement learning (MARL) in LLM collaboration. However, constructing reliable rewards is difficult in practice, as complete and accurate metrics are often unavailable and hard to aggregate. Preference learning provides an alternative by learning from comparative human or AI feedback. Yet, its extension to multi-agent systems remains underexplored. To address this gap, we formulate preference-based multi-agent systems (MAS) from decentralized and centralized collaboration perspectives. We also introduce a general multi-agent preference learning framework (MAPL) to solve these problems. MAPL allows iterative updates by comparing the current solution with decentralized or centralized solutions generated by various agents. We instantiate MAPL using MARL from human feedback (MARLHF) with a learned reward model and multi-agent direct preference optimization (MADPO). Experiments on collaborative writing, coding, tool use, and travel planning show that MAPL can improve collaboration quality and efficiency while approaching the performance of MARL with fixed, well-defined rewards. Within MAPL, MARLHF generally outperforms MADPO on most tasks but remains sensitive to data coverage, agent and comparator models, and the underlying MARL algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 59 of its 77 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

59 of 77 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic29 Jan
  2. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs13 Oct
  3. 2025LLM Collaboration With Multi-Agent Reinforcement Learning6 Aug
  4. 2025MARFT: Multi-Agent Reinforcement Fine-Tuning21 Apr
  5. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning25 Feb
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 52

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
38 more
  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  2. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  6. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  7. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  12. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  13. 2019The StarCraft Multi-Agent Challengecited by 597
  14. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  15. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  16. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  17. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  18. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  19. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 333
  20. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 309
  21. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  22. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  23. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 98
  24. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  25. 2025AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systemscited by 90
  26. 2024Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligencecited by 86
  27. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 79
  28. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  29. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 67
  30. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 61
  31. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  32. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 50
  33. 2025Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learningcited by 39
  34. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 37
  35. 2023A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problemcited by 19
  36. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  37. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  38. 2025O-MAPL: Offline Multi-agent Preference Learningcited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization
  2. 2025Enhancing Multi-Agent Systems via Reinforcement Learning with LLM-based Planner and Graph-based Policy
  3. 2025Training Proactive and Personalized LLM Agents
  4. 2026Multi-Agent Reinforcement Learning via Agent-Specific Preference
  5. 2026Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning
  6. 2024PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models ↗
  7. 2026Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards
  8. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learning