# Deploying Foundation Model-Enabled Air and Ground Robots in the Field: Challenges and Opportunities

arXiv: https://arxiv.org/abs/2505.09477 · HTML: https://mlsanity.com/paper/2505.09477/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 39 references that are in the map.
- First posted on arXiv 14 May 2025.

## Builds on (most cited first, 30 of 33)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [AerialVLN: Vision-and-Language Navigation for UAVs](https://mlsanity.com/paper/2308.06735/) (2023)
- [Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning](https://mlsanity.com/paper/2310.10103/) (2023)
- [HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigation](https://mlsanity.com/paper/2409.14296/) (2024)
- [Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulation](https://mlsanity.com/paper/2403.08605/) (2024)
- [ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints](https://mlsanity.com/paper/2202.11271/) (2022)
- [Driving Everywhere with Large Language Model Policy Adaptation](https://mlsanity.com/paper/2402.05932/) (2024)
- [Deploying and Evaluating LLMs to Program Service Mobile Robots](https://mlsanity.com/paper/2311.11183/) (2023)
- [NeBula: Quest for Robotic Autonomy in Challenging Environments; TEAM CoSTAR at the DARPA Subterranean Challenge](https://mlsanity.com/paper/2103.11470/) (2021)
- [RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigation](https://mlsanity.com/paper/2405.05792/) (2024)
- [Exploring Spatial Representation to Enhance LLM Reasoning in Aerial Vision-Language Navigation](https://mlsanity.com/paper/2410.08500/) (2024)
- [SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environments](https://mlsanity.com/paper/2410.03035/) (2024)
- [GroundGrid:LiDAR Point Cloud Ground Segmentation and Terrain Estimation](https://mlsanity.com/paper/2405.15664/) (2024)
- [Reasoning about the Unseen for Efficient Outdoor Object Navigation](https://mlsanity.com/paper/2309.10103/) (2023)
- [Large-scale Autonomous Flight with Real-time Semantic SLAM under Dense Forest Canopy](https://mlsanity.com/paper/2109.06479/) (2021)
- [Stronger Together: Air-Ground Robotic Collaboration Using Semantics](https://mlsanity.com/paper/2206.14289/) (2022)
- [NEUSIS: A Compositional Neuro-Symbolic Framework for Autonomous Perception, Reasoning, and Planning in Complex UAV Search Missions](https://mlsanity.com/paper/2409.10196/) (2024)

## Built on by (most cited first, 3 of 3)

- [Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligence](https://mlsanity.com/paper/2510.26915/) (2025)
- [CoFL: Continuous Flow Fields for Language-Conditioned Navigation](https://mlsanity.com/paper/2603.02854/) (2026)
- [CoFineLLM: Conformal Finetuning of LLMs for Language-Instructed Robot Planning](https://mlsanity.com/paper/2511.06575/) (2025)

## Abstract (arXiv)

The integration of foundation models (FMs) into robotics has enabled robots to understand natural language and reason about the semantics in their environments. However, existing FM-enabled robots primary operate in closed-world settings, where the robot is given a full prior map or has a full view of its workspace. This paper addresses the deployment of FM-enabled robots in the field, where missions often require a robot to operate in large-scale and unstructured environments. To effectively accomplish these missions, robots must actively explore their environments, navigate obstacle-cluttered terrain, handle unexpected sensor inputs, and operate with compute constraints. We discuss recent deployments of SPINE, our LLM-enabled autonomy framework, in field robotic settings. To the best of our knowledge, we present the first demonstration of large-scale LLM-enabled robot planning in unstructured environments with several kilometers of missions. SPINE is agnostic to a particular LLM, which allows us to distill small language models capable of running onboard size, weight and power (SWaP) limited platforms. Via preliminary model distillation work, we then present the first language-driven UAV planner using on-device language models. We conclude our paper by proposing several promising directions for future research.