# Air-Ground Collaboration for Language-Specified Missions in Unknown Environments

arXiv: https://arxiv.org/abs/2505.09108 · HTML: https://mlsanity.com/paper/2505.09108/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 97 references that are in the map.
- First posted on arXiv 14 May 2025.

## Builds on (most cited first, 30 of 58)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2307.04738/) (2023)
- [Open-vocabulary Queryable Scene Representations for Real World Planning](https://mlsanity.com/paper/2209.09874/) (2022)
- [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023)
- [AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers](https://mlsanity.com/paper/2306.06531/) (2023)
- [Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?](https://mlsanity.com/paper/2309.15943/) (2023)
- [On-Manifold Preintegration for Real-Time Visual-Inertial Odometry](https://mlsanity.com/paper/1512.02363/) (2015)
- [Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning](https://mlsanity.com/paper/2310.10103/) (2023)
- [DrEureka: Language Model Guided Sim-To-Real Transfer](https://mlsanity.com/paper/2406.01967/) (2024)
- [Clio: Real-time Task-Driven Open-Set 3D Scene Graphs](https://mlsanity.com/paper/2404.13696/) (2024)
- [Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs](https://mlsanity.com/paper/2407.07775/) (2024)
- [Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulation](https://mlsanity.com/paper/2403.08605/) (2024)
- [Real-Time Anomaly Detection and Reactive Planning with Large Language Models](https://mlsanity.com/paper/2407.08735/) (2024)
- [Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-time Systems](https://mlsanity.com/paper/2305.07154/) (2023)

## Built on by (most cited first, 11 of 11)

- [Large Language Model-Assisted UAV Operations and Communications: A Multifaceted Survey and Tutorial](https://mlsanity.com/paper/2602.19534/) (2026)
- [Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligence](https://mlsanity.com/paper/2510.26915/) (2025)
- [MemoGuard: An Adaptive Runtime for Guarding Against Memory Traps in Communication-Limited Robot Navigation](https://mlsanity.com/paper/2607.15589/) (2026)
- [AerialClaw: An Open-Source Framework for LLM-Driven Autonomous Aerial Agents](https://mlsanity.com/paper/2606.12142/) (2026)
- [Adaptive Per-Tree Canopy Volume Estimation Using Mobile LiDAR in Structured and Unstructured Orchards](https://mlsanity.com/paper/2506.08061/) (2025)
- [Prompting Robot Teams with Natural Language](https://mlsanity.com/paper/2509.24575/) (2025)
- [LMPath: Language-Mediated Priors and Path Generation for Aerial Exploration](https://mlsanity.com/paper/2605.13782/) (2026)
- [Co-GLANCE: Uncertainty-Aware Active Perception for Heterogeneous Robot Teaming](https://mlsanity.com/paper/2606.09919/) (2026)
- [UDAV: Uncertainty-Driven Adaptive VLM Waypoint Planner](https://mlsanity.com/paper/2609.16368/) (2026)
- [HEROIC: Heterogeneous Evidential Reasoning for Open-Vocabulary Identification and Cross-Robot Collaboration](https://mlsanity.com/paper/2609.19803/) (2026)
- [FRESHLATENT: Channel-Aware Latent Adaptation for Resource-Constrained Embodied VLM Perception](https://mlsanity.com/paper/2609.30629/) (2026)

## Abstract (arXiv)

As autonomous robotic systems become increasingly mature, users will want to specify missions at the level of intent rather than in low-level detail. Language is an expressive and intuitive medium for such mission specification. However, realizing language-guided robotic teams requires overcoming significant technical hurdles. Interpreting and realizing language-specified missions requires advanced semantic reasoning. Successful heterogeneous robots must effectively coordinate actions and share information across varying viewpoints. Additionally, communication between robots is typically intermittent, necessitating robust strategies that leverage communication opportunities to maintain coordination and achieve mission objectives. In this work, we present a first-of-its-kind system where an unmanned aerial vehicle (UAV) and an unmanned ground vehicle (UGV) are able to collaboratively accomplish missions specified in natural language while reacting to changes in specification on the fly. We leverage a Large Language Model (LLM)-enabled planner to reason over semantic-metric maps that are built online and opportunistically shared between an aerial and a ground robot. We consider task-driven navigation in urban and rural areas. Our system must infer mission-relevant semantics and actively acquire information via semantic mapping. In both ground and air-ground teaming experiments, we demonstrate our system on seven different natural-language specifications at up to kilometer-scale navigation.