mlsanityData to 14 Oct 2026

Position: Beyond Training Time, Test-Time Coordination is Essential for Cooperative MARL

arXiv 2610.17409 ↗First posted 12 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We argue that cooperative multi-agent reinforcement learning (MARL) should treat test-time coordination as part of the problem definition rather than a deployment artifact. Our focus is on long-horizon, centrally monitored cooperative systems such as warehouse robotics, automated storage-and-retrieval systems, and cloud-resource scheduling, where teams share a reward, operate under shifting objectives, and already run with global telemetry at deployment. The dominant paradigm in this regime, centralized training with decentralized execution (CTDE), uses global information only during training and provides no coordination channel at execution. This is sufficient when agents are weakly coupled or the environment is stationary, but it fails when the team must realign after conditions change. The limitation is structural rather than algorithmic, and more training cannot add a channel that does not exist at execution time. We call this the recoverability gap. To address it, we name the underexplored design space of intermittent test-time coordination as centralized training with mixed execution (CTME), in which a coordinator observes global state and emits high-level signals while agents act locally. CTME is not a particular learning algorithm, but a cooperative MARL problem setting that makes test-time information structure explicit. It specifies team-level information, high-level signals, and update frequency after training, with CTDE recovered as a limit case. We do not argue that CTDE is obsolete. We argue that removing the test-time channel is a modeling choice, not a necessity, and the wrong default when global telemetry is available and teams must recover from drift.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 54 of its 90 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

54 of 90 references.

References · 54

  1. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  4. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  6. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  7. 2019The StarCraft Multi-Agent Challengecited by 597
  8. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 575
  9. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  10. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  12. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
38 more
  1. 2016Learning Multiagent Communication with Backpropagationcited by 366
  2. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  3. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  4. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  5. 2018TarMAC: Targeted Multi-Agent Communicationcited by 208
  6. 2018Mean Field Multi-Agent Reinforcement Learningcited by 207
  7. 2018Learning Attentional Communication for Multi-Agent Cooperationcited by 175
  8. 2019Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Controlcited by 164
  9. 2019CoLight: Learning Network-level Cooperation for Traffic Signal Controlcited by 138
  10. 2018Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Taskscited by 121
  11. 2021Learning One Representation to Optimize All Rewardscited by 115
  12. 2017Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Gamescited by 96
  13. 2017A Survey of Learning in Multiagent Environments: Dealing with Non-Stationaritycited by 94
  14. 2019Learning to Schedule Communication in Multi-agent Reinforcement Learningcited by 93
  15. 2019Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learningcited by 93
  16. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  17. 2019On the Pitfalls of Measuring Emergent Communicationcited by 71
  18. 2020Age of Information: An Introduction and Surveycited by 70
  19. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 68
  20. 2017VAIN: Attentional Multi-agent Predictive Modelingcited by 53
  21. 2019Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discoverycited by 43
  22. 2019Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learningcited by 33
  23. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 33
  24. 2021Continuous Coordination As a Realistic Scenario for Lifelong Learningcited by 32
  25. 2022Scalable Multi-Agent Reinforcement Learning for Warehouse Logistics with Robotic and Human Co-Workerscited by 30
  26. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 24
  27. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 20
  28. 2022ALMA: Hierarchical Learning for Composite Multi-Agent Taskscited by 18
  29. 2021HAMMER: Multi-Level Coordination of Reinforcement Learning Agents via Learned Messagingcited by 13
  30. 2024Hierarchical Consensus-Based Multi-Agent Reinforcement Learning for Multi-Robot Cooperation Taskscited by 13
  31. 2017Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learningcited by 12
  32. 2024IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learningcited by 7
  33. 2024Neural Coordination and Capacity Control for Inventory Managementcited by 6
  34. 2025Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learningcited by 6
  35. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 6
  36. 2024Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learningcited by 5
  37. 2025Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systemscited by 5
  38. 2026DeepStock: Reinforcement Learning with Policy Regularizations for Inventory Managementcited by 5

50 most cited of 54 All 54 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Coordination Matters: Evaluation of Cooperative Multi-Agent Reinforcement Learning
  2. 2024An Initial Introduction to Cooperative Multi-Agent Reinforcement Learning
  3. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning
  4. 2025Remembering the Markov Property in Cooperative MARL
  5. 2025An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks
  6. 2025CTC: The Composite Task Challenge for Cooperative Multi-Agent Reinforcement Learning
  7. 2025Optimization without Future Compromises? Decentralized Coordination via Collective and Reinforcement Learning
  8. 2025Multi-agent In-context Coordination via Decentralized Memory Retrieval