# Controlling Collectives of AI Agents in Reasoning Space with Spatial Transformers

arXiv: https://arxiv.org/abs/2609.28247 · HTML: https://mlsanity.com/paper/2609.28247/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 24 of its 25 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 24 of 24)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2307.04738/) (2023)
- [Physically Grounded Vision-Language Models for Robotic Manipulation](https://mlsanity.com/paper/2309.02561/) (2023)
- [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023)
- [Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?](https://mlsanity.com/paper/2309.15943/) (2023)
- [Embodied LLM Agents Learn to Cooperate in Organized Teams](https://mlsanity.com/paper/2403.12482/) (2024)
- [Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation](https://mlsanity.com/paper/2303.03480/) (2023)
- [Learning Decentralized Controllers for Robot Swarms with Graph Neural Networks](https://mlsanity.com/paper/1903.10527/) (2019)
- [Language Models as Zero-Shot Trajectory Generators](https://mlsanity.com/paper/2310.11604/) (2023)
- [COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models](https://mlsanity.com/paper/2409.15146/) (2024)
- [Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity](https://mlsanity.com/paper/2404.03570/) (2024)
- [MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulation](https://mlsanity.com/paper/2411.17636/) (2024)
- [SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environments](https://mlsanity.com/paper/2410.03035/) (2024)
- [I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences](https://mlsanity.com/paper/2411.12960/) (2024)
- [Distilling On-device Language Models for Robot Planning with Minimal Human Intervention](https://mlsanity.com/paper/2506.17486/) (2025)
- [ZeroCAP: Zero-Shot Multi-Robot Context Aware Pattern Formation via Large Language Models](https://mlsanity.com/paper/2404.02318/) (2024)
- [LPAC: Learnable Perception-Action-Communication Loops with Applications to Coverage Control](https://mlsanity.com/paper/2401.04855/) (2024)
- [LLM-Flock: Decentralized Multi-Robot Flocking via Large Language Models and Influence-Based Consensus](https://mlsanity.com/paper/2505.06513/) (2025)
- [MAST: Multi-Agent Spatial Transformer for Learning to Collaborate](https://mlsanity.com/paper/2509.17195/) (2025)
- [Ro-SLM: Onboard Small Language Models for Robot Task Planning and Operation Code Generation](https://mlsanity.com/paper/2604.10929/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) introduce an exciting new paradigm for planning and navigation in robotics, but fail on even simple multi-robot tasks as team sizes grow. We propose COMPASS, a scalable, decentralized multi-robot architecture for controlling large collectives of agentic robots with reasoning space feedback control. Feedback is generated locally on each robot by a spatial transformer which aggregates multi-hop messages across the fleet into a learned feedback token. Our experiments find that collectives of language models demonstrate performance gains from structured diversity of the input command, which can cancel biases; an advantage that is held across scale. Compared against a centralized frontier LLM policy and a language-only communication ablation, we find that the coupled design of COMPASS decisively produces cohesive flocking formations that accurately fly the commanded intent. We show that reasoning feedback works best when composed with a compact learned token. Our ablations show that hand engineered feedback with raw state appearing in the language channel obliterates cohesion. COMPASS generalizes zero-shot to unseen instructions of ambiguous meaning while commanding flocks up to 16 times its training scale, flying up to 1024 robots under natural language commands.