# ICCO: Learning an Instruction-conditioned Coordinator for Language-guided Task-aligned Multi-robot Control

arXiv: https://arxiv.org/abs/2503.12122 · HTML: https://mlsanity.com/paper/2503.12122/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 15 Mar 2025.

## Builds on (most cited first, 23 of 23)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.08839/) (2020)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2307.04738/) (2023)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023)
- [Task and Motion Planning with Large Language Models for Object Rearrangement](https://mlsanity.com/paper/2303.06247/) (2023)
- [AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers](https://mlsanity.com/paper/2306.06531/) (2023)
- [Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?](https://mlsanity.com/paper/2309.15943/) (2023)
- [Multi-Agent Common Knowledge Reinforcement Learning](https://mlsanity.com/paper/1810.11702/) (2018)
- [Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition](https://mlsanity.com/paper/2105.08692/) (2021)
- [LiP-LLM: Integrating Linear Programming and dependency graph with Large Language Models for multi-robot task planning](https://mlsanity.com/paper/2410.21040/) (2024)
- [Kimera-Multi: a System for Distributed Multi-Robot Metric-Semantic Simultaneous Localization and Mapping](https://mlsanity.com/paper/2011.04087/) (2020)
- [Probabilistically Correct Language-based Multi-Robot Planning using Conformal Prediction](https://mlsanity.com/paper/2402.15368/) (2024)
- [Multi-Robot Cooperative Socially-Aware Navigation Using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2309.15234/) (2023)
- [Reinforcement Learned Distributed Multi-Robot Navigation with Reciprocal Velocity Obstacle Shaped Rewards](https://mlsanity.com/paper/2203.10229/) (2022)
- [Resilient and Distributed Multi-Robot Visual SLAM: Datasets, Experiments, and Lessons Learned](https://mlsanity.com/paper/2304.04362/) (2023)
- [Large Language Models for Orchestrating Bimanual Robots](https://mlsanity.com/paper/2404.02018/) (2024)
- [Language-Conditioned Offline RL for Multi-Robot Navigation](https://mlsanity.com/paper/2407.20164/) (2024)
- [Cooperative Grasping and Transportation using Multi-agent Reinforcement Learning with Ternary Force Representation](https://mlsanity.com/paper/2411.13942/) (2024)

## Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have permitted the development of language-guided multi-robot systems, which allow robots to execute tasks based on natural language instructions. However, achieving effective coordination in distributed multi-agent environments remains challenging due to (1) misalignment between instructions and task requirements and (2) inconsistency in robot behaviors when they independently interpret ambiguous instructions. To address these challenges, we propose Instruction-Conditioned Coordinator (ICCO), a Multi-Agent Reinforcement Learning (MARL) framework designed to enhance coordination in language-guided multi-robot systems. ICCO consists of a Coordinator agent and multiple Local Agents, where the Coordinator generates Task-Aligned and Consistent Instructions (TACI) by integrating language instructions with environmental states, ensuring task alignment and behavioral consistency. The Coordinator and Local Agents are jointly trained to optimize a reward function that balances task efficiency and instruction following. A Consistency Enhancement Term is added to the learning objective to maximize mutual information between instructions and robot behaviors, further improving coordination. Simulation and real-world experiments validate the effectiveness of ICCO in achieving language-guided task-aligned multi-robot control. The demonstration can be found at https://yanoyoshiki.github.io/ICCO/.