# Leveraging Large Language Model for Heterogeneous Ad Hoc Teamwork Collaboration

arXiv: https://arxiv.org/abs/2406.12224 · HTML: https://mlsanity.com/paper/2406.12224/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2024.

## Built on by (most cited first, 7 of 7)

- [COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models](https://mlsanity.com/paper/2409.15146/) (2024)
- [Orchestrating Human-AI Teams: The Manager Agent as a Unifying Research Challenge](https://mlsanity.com/paper/2510.02557/) (2025)
- [VernaCopter: Disambiguated Natural-Language-Driven Robot via Formal Specifications](https://mlsanity.com/paper/2409.09536/) (2024)
- [LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability](https://mlsanity.com/paper/2607.06157/) (2026)
- [Leveraging Adaptive Group Negotiation for Heterogeneous Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2602.06967/) (2025)
- [A Closed-Loop Multi-Agent Framework for Robust Multi-Robot Manipulation](https://mlsanity.com/paper/2607.06990/) (2026)
- [Learning When to Cooperate Under Heterogeneous Goals](https://mlsanity.com/paper/2603.07253/) (2026)

## Abstract (arXiv)

Compared with the widely investigated homogeneous multi-robot collaboration, heterogeneous robots with different capabilities can provide a more efficient and flexible collaboration for more complex tasks. In this paper, we consider a more challenging heterogeneous ad hoc teamwork collaboration problem where an ad hoc robot joins an existing heterogeneous team for a shared goal. Specifically, the ad hoc robot collaborates with unknown teammates without prior coordination, and it is expected to generate an appropriate cooperation policy to improve the efficiency of the whole team. To solve this challenging problem, we leverage the remarkable potential of the large language model (LLM) to establish a decentralized heterogeneous ad hoc teamwork collaboration framework that focuses on generating reasonable policy for an ad hoc robot to collaborate with original heterogeneous teammates. A training-free hierarchical dynamic planner is developed using the LLM together with the newly proposed Interactive Reflection of Thoughts (IRoT) method for the ad hoc agent to adapt to different teams. We also build a benchmark testing dataset to evaluate the proposed framework in the heterogeneous ad hoc multi-agent tidying-up task. Extensive comparison and ablation experiments are conducted in the benchmark to demonstrate the effectiveness of the proposed framework. We have also employed the proposed framework in physical robots in a real-world scenario. The experimental videos can be found at https://youtu.be/wHYP5T2WIp0.