mlsanityData to 14 Oct 2026

Air-Ground Collaboration for Language-Specified Missions in Unknown Environments

arXiv 2505.09108 ↗First posted 14 May 2025cs.RO cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

As autonomous robotic systems become increasingly mature, users will want to specify missions at the level of intent rather than in low-level detail. Language is an expressive and intuitive medium for such mission specification. However, realizing language-guided robotic teams requires overcoming significant technical hurdles. Interpreting and realizing language-specified missions requires advanced semantic reasoning. Successful heterogeneous robots must effectively coordinate actions and share information across varying viewpoints. Additionally, communication between robots is typically intermittent, necessitating robust strategies that leverage communication opportunities to maintain coordination and achieve mission objectives. In this work, we present a first-of-its-kind system where an unmanned aerial vehicle (UAV) and an unmanned ground vehicle (UGV) are able to collaboratively accomplish missions specified in natural language while reacting to changes in specification on the fly. We leverage a Large Language Model (LLM)-enabled planner to reason over semantic-metric maps that are built online and opportunistically shared between an aerial and a ground robot. We consider task-driven navigation in urban and rural areas. Our system must infer mission-relevant semantics and actively acquire information via semantic mapping. In both ground and air-ground teaming experiments, we demonstrate our system on seven different natural-language specifications at up to kilometer-scale navigation.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 58 of its 97 references that are in the map. First posted on arXiv 14 May 2025.

Builds on

58 of 97 references.

References · 58

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  10. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  11. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  12. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
38 more
  1. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  2. 2022Visual Language Maps for Robot Navigationcited by 365
  3. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  4. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 294
  5. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  6. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  7. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  8. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 153
  9. 2023AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkerscited by 113
  10. 2023Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?cited by 106
  11. 2015On-Manifold Preintegration for Real-Time Visual-Inertial Odometrycited by 95
  12. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 89
  13. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 83
  14. 2024Clio: Real-time Task-Driven Open-Set 3D Scene Graphscited by 81
  15. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 73
  16. 2024Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulationcited by 70
  17. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 68
  18. 2023Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-time Systemscited by 60
  19. 2023NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Modelscited by 59
  20. 2023LLM-Assist: Enhancing Closed-Loop Planning with Language-Based Reasoningcited by 59
  21. 2021Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systemscited by 54
  22. 2023Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environmentscited by 53
  23. 2024COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Modelscited by 45
  24. 2023Deploying and Evaluating LLMs to Program Service Mobile Robotscited by 36
  25. 2021NeBula: Quest for Robotic Autonomy in Challenging Environments; TEAM CoSTAR at the DARPA Subterranean Challengecited by 32
  26. 2023Optimal Scene Graph Planning with Large Language Model Guidancecited by 27
  27. 2023Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologiescited by 25
  28. 2024RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigationcited by 25
  29. 2023Hydra-Multi: Collaborative Online Construction of 3D Scene Graphs with Multi-Robot Teamscited by 21
  30. 2023SlimSAM: 0.1% Data Makes Segment Anything Slimcited by 20
  31. 2023REAL: Resilience and Adaptation using Large Language Models on Autonomous Aerial Robotscited by 19
  32. 2024SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environmentscited by 18
  33. 2021Semantic OcTree Mapping and Shannon Mutual Information Computation for Robot Explorationcited by 16
  34. 2024Verifiably Following Complex Robot Instructions with Foundation Modelscited by 16
  35. 2024GroundGrid:LiDAR Point Cloud Ground Segmentation and Terrain Estimationcited by 14
  36. 2023Reasoning about the Unseen for Efficient Outdoor Object Navigationcited by 13
  37. 2021Large-scale Autonomous Flight with Real-time Semantic SLAM under Dense Forest Canopycited by 12
  38. 2019Mine Tunnel Exploration using Multiple Quadrupedal Robotscited by 8

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2026Large Language Model-Assisted UAV Operations and Communications: A Multifaceted Survey and Tutorialcited by 4
  2. 2025Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligencecited by 3
  3. 2026MemoGuard: An Adaptive Runtime for Guarding Against Memory Traps in Communication-Limited Robot Navigationcited by 2
  4. 2026AerialClaw: An Open-Source Framework for LLM-Driven Autonomous Aerial Agentscited by 1
  5. 2025Adaptive Per-Tree Canopy Volume Estimation Using Mobile LiDAR in Structured and Unstructured Orchardscited by 0
  6. 2025Prompting Robot Teams with Natural Languagecited by 0
  7. 2026LMPath: Language-Mediated Priors and Path Generation for Aerial Explorationcited by 0
  8. 2026Co-GLANCE: Uncertainty-Aware Active Perception for Heterogeneous Robot Teamingcited by 0
  9. 2026UDAV: Uncertainty-Driven Adaptive VLM Waypoint Plannercited by 0
  10. 2026HEROIC: Heterogeneous Evidential Reasoning for Open-Vocabulary Identification and Cross-Robot Collaborationcited by 0
  11. 2026FRESHLATENT: Channel-Aware Latent Adaptation for Resource-Constrained Embodied VLM Perceptioncited by 0