# ZeroCAP: Zero-Shot Multi-Robot Context Aware Pattern Formation via Large Language Models

arXiv: https://arxiv.org/abs/2404.02318 · HTML: https://mlsanity.com/paper/2404.02318/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2024.

## Built on by (most cited first, 8 of 8)

- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [Large Language Models for Multi-Robot Systems: A Survey](https://mlsanity.com/paper/2502.03814/) (2025)
- [GenSwarm: Scalable Multi-Robot Code-Policy Generation and Deployment via Language Models](https://mlsanity.com/paper/2503.23875/) (2025)
- [Can a Robot Walk the Robotic Dog: Triple-Zero Collaborative Navigation for Heterogeneous Multi-Agent Systems](https://mlsanity.com/paper/2603.21723/) (2026)
- [Prompting Robot Teams with Natural Language](https://mlsanity.com/paper/2509.24575/) (2025)
- [ConPoSe: LLM-Guided Contact Point Selection for Scalable Cooperative Object Pushing](https://mlsanity.com/paper/2510.08705/) (2025)
- [Shape Formation for the Cooperative Transportation of Arbitrary Objects Using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2606.09610/) (2026)
- [Controlling Collectives of AI Agents in Reasoning Space with Spatial Transformers](https://mlsanity.com/paper/2609.28247/) (2026)

## Abstract (arXiv)

Incorporating language comprehension into robotic operations unlocks significant advancements in robotics, but also presents distinct challenges, particularly in executing spatially oriented tasks like pattern formation. This paper introduces ZeroCAP, a novel system that integrates large language models with multi-robot systems for zero-shot context aware pattern formation. Grounded in the principles of language-conditioned robotics, ZeroCAP leverages the interpretative power of language models to translate natural language instructions into actionable robotic configurations. This approach combines the synergy of vision-language models, cutting-edge segmentation techniques and shape descriptors, enabling the realization of complex, context-driven pattern formations in the realm of multi robot coordination. Through extensive experiments, we demonstrate the systems proficiency in executing complex context aware pattern formations across a spectrum of tasks, from surrounding and caging objects to infilling regions. This not only validates the system's capability to interpret and implement intricate context-driven tasks but also underscores its adaptability and effectiveness across varied environments and scenarios. The experimental videos and additional information about this work can be found at https://sites.google.com/view/zerocap/home.