# Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligence arXiv: https://arxiv.org/abs/2510.26915 ยท HTML: https://mlsanity.com/paper/2510.26915/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 28 of its 48 references that are in the map. - First posted on arXiv 30 Oct 2025. ## Placed in - Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/) ## Builds on (most cited first, 28 of 28) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024) - [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2307.04738/) (2023) - [ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://mlsanity.com/paper/2312.00784/) (2023) - [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023) - [Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?](https://mlsanity.com/paper/2309.15943/) (2023) - [Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems](https://mlsanity.com/paper/2106.14386/) (2021) - [COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models](https://mlsanity.com/paper/2409.15146/) (2024) - [Direct LiDAR-Inertial Odometry: Lightweight LIO with Continuous-Time Motion Correction](https://mlsanity.com/paper/2203.03749/) (2022) - [SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environments](https://mlsanity.com/paper/2410.03035/) (2024) - [A Critical Review of Communications in Multi-Robot Systems](https://mlsanity.com/paper/2206.09484/) (2022) - [LAMP 2.0: A Robust Multi-Robot SLAM System for Operation in Challenging Large-Scale Underground Environments](https://mlsanity.com/paper/2205.13135/) (2022) - [GroundGrid:LiDAR Point Cloud Ground Segmentation and Terrain Estimation](https://mlsanity.com/paper/2405.15664/) (2024) - [Air-Ground Collaboration for Language-Specified Missions in Unknown Environments](https://mlsanity.com/paper/2505.09108/) (2025) - [Distilling On-device Language Models for Robot Planning with Minimal Human Intervention](https://mlsanity.com/paper/2506.17486/) (2025) - [Generalized Mission Planning for Heterogeneous Multi-Robot Teams via LLM-constructed Hierarchical Trees](https://mlsanity.com/paper/2501.16539/) (2025) - [Mine Tunnel Exploration using Multiple Quadrupedal Robots](https://mlsanity.com/paper/1909.09662/) (2019) - [Multi-robot Mission Planning in Dynamic Semantic Environments](https://mlsanity.com/paper/2209.06323/) (2022) - [Language-Grounded Hierarchical Planning and Execution with Multi-Robot 3D Scene Graphs](https://mlsanity.com/paper/2506.07454/) (2025) - [Air-Ground Collaboration with SPOMP: Semantic Panoramic Online Mapping and Planning](https://mlsanity.com/paper/2407.09902/) (2024) - [Enabling Large-scale Heterogeneous Collaboration with Opportunistic Communications](https://mlsanity.com/paper/2309.15975/) (2023) - [ATLAS Navigator: Active Task-driven LAnguage-embedded Gaussian Splatting](https://mlsanity.com/paper/2502.20386/) (2025) - [Multi-Robot Coordination and Cooperation with Task Precedence Relationships](https://mlsanity.com/paper/2209.14417/) (2022) - [Deploying Foundation Model-Enabled Air and Ground Robots in the Field: Challenges and Opportunities](https://mlsanity.com/paper/2505.09477/) (2025) - [DEXTER-LLM: Dynamic and Explainable Coordination of Multi-Robot Systems in Unknown Environments via Large Language Models](https://mlsanity.com/paper/2508.14387/) (2025) - [Challenges and Opportunities for Large-Scale Exploration with Air-Ground Teams using Semantics](https://mlsanity.com/paper/2405.07169/) (2024) - [Capability-aware Task Allocation and Team Formation Analysis for Cooperative Exploration of Complex Environments](https://mlsanity.com/paper/2411.00400/) (2024) ## Built on by (most cited first, 3 of 3) - [MemoGuard: An Adaptive Runtime for Guarding Against Memory Traps in Communication-Limited Robot Navigation](https://mlsanity.com/paper/2607.15589/) (2026) - [Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language](https://mlsanity.com/paper/2609.30428/) (2026) - [Co-GLANCE: Uncertainty-Aware Active Perception for Heterogeneous Robot Teaming](https://mlsanity.com/paper/2606.09919/) (2026) ## Abstract (arXiv) While heterogeneous teams have typically been designed for well-specified missions with known semantics, generative intelligence, i.e., large language models (LLMs) and vision language models (VLMs), opens the possibility of teams that infer mission-relevant semantics and subtasks given high-level natural language specifications and environmental context. However, current LLM- and VLM-enabled teaming methods typically assume well-structured and known environments, limiting performance in complex real-world settings. We address these limitations via SPINE-HT, a framework that grounds the reasoning abilities of LLMs in the evolving context of a heterogeneous robot team through a three-stage process. Given mission specifications and team capabilities in natural language, an LLM infers necessary subtasks. These subtasks are validated for feasibility and then assigned to robots based on capabilities such as traversability or perception. Finally, the framework uses feedback (semantic maps and task outcomes) acquired by the robots to update subtasks online. In simulation experiments with closed-loop perception and control, our framework achieves nearly twice the success rate compared to prior LLM-enabled heterogeneous teaming approaches. In real-world experiments with a Clearpath Jackal, a Clearpath Husky, a Boston Dynamics Spot, and a high-altitude UAV, our method achieves an 87.5% success rate in missions requiring reasoning about robot capabilities and refining subtasks with online feedback. More information is provided at: https://zacravichandran.github.io/SPINE-HT.