# Deploying and Evaluating LLMs to Program Service Mobile Robots

arXiv: https://arxiv.org/abs/2311.11183 · HTML: https://mlsanity.com/paper/2311.11183/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2023.

## Built on by (most cited first, 30 of 36)

- [ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation](https://mlsanity.com/paper/2409.13682/) (2024)
- [Safety Guardrails for LLM-Enabled Robots](https://mlsanity.com/paper/2503.07885/) (2025)
- [Adversarial Attacks on Robotic Vision Language Action Models](https://mlsanity.com/paper/2506.03350/) (2025)
- [RHO: Your Coding Agent is Secretly a Roboticist](https://mlsanity.com/paper/2606.16458/) (2026)
- [SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environments](https://mlsanity.com/paper/2410.03035/) (2024)
- [Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art](https://mlsanity.com/paper/2403.16527/) (2024)
- [VoicePilot: Harnessing LLMs as Speech Interfaces for Physically Assistive Robots](https://mlsanity.com/paper/2404.04066/) (2024)
- [Air-Ground Collaboration for Language-Specified Missions in Unknown Environments](https://mlsanity.com/paper/2505.09108/) (2025)
- [Distilling On-device Language Models for Robot Planning with Minimal Human Intervention](https://mlsanity.com/paper/2506.17486/) (2025)
- [Language Models can Infer Action Semantics for Symbolic Planners from Environment Feedback](https://mlsanity.com/paper/2406.02791/) (2024)
- [Safety Control of Service Robots with LLMs and Embodied Knowledge Graphs](https://mlsanity.com/paper/2405.17846/) (2024)
- [Contrast Sets for Evaluating Language-Guided Robot Policies](https://mlsanity.com/paper/2406.13636/) (2024)
- [Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selection](https://mlsanity.com/paper/2502.09829/) (2025)
- [Robo-Instruct: Simulator-Augmented Instruction Alignment For Finetuning Code LLMs](https://mlsanity.com/paper/2405.20179/) (2024)
- [Joint Verification and Refinement of Language Models for Safety-Constrained Planning](https://mlsanity.com/paper/2410.14865/) (2024)
- [Understanding How CodeLLMs (Mis)Predict Types with Activation Steering](https://mlsanity.com/paper/2404.01903/) (2024)
- [Industry 6.0: New Generation of Industry driven by Generative AI and Swarm of Heterogeneous Robots](https://mlsanity.com/paper/2409.10106/) (2024)
- [Explainable Behavior Cloning: Teaching Large Language Model Agents through Learning by Demonstration](https://mlsanity.com/paper/2410.22916/) (2024)
- [A Survey on AgentOps: Categorization, Challenges, and Future Directions](https://mlsanity.com/paper/2508.02121/) (2025)
- [ComposableNav: Instruction-Following Navigation in Dynamic Environments via Composable Diffusion](https://mlsanity.com/paper/2509.17941/) (2025)
- [Deploying Foundation Model-Enabled Air and Ground Robots in the Field: Challenges and Opportunities](https://mlsanity.com/paper/2505.09477/) (2025)
- [VASO: Formally Verifiable Self-Evolving Skills for Physical AI Agents](https://mlsanity.com/paper/2606.05395/) (2026)
- [How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study](https://mlsanity.com/paper/2604.06750/) (2026)
- [SYNAPSE: SYmbolic Neural-Aided Preference Synthesis Engine](https://mlsanity.com/paper/2403.16689/) (2024)
- [Creating and Repairing Robot Programs in Open-World Domains](https://mlsanity.com/paper/2410.18893/) (2024)
- [Characterizing the Robustness of Black-Box LLM Planners Under Perturbed Observations with Adaptive Stress Testing](https://mlsanity.com/paper/2505.05665/) (2025)
- [Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models](https://mlsanity.com/paper/2506.14727/) (2025)
- [PSALM-V: Automating Symbolic Planning in Interactive Visual Environments with Large Language Models](https://mlsanity.com/paper/2506.20097/) (2025)
- [Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Representation](https://mlsanity.com/paper/2408.08188/) (2024)
- [Language-guided Robust Navigation for Mobile Robots in Dynamically-changing Environments](https://mlsanity.com/paper/2409.19459/) (2024)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) have spurred interest in using them for generating robot programs from natural language, with promising initial results. We investigate the use of LLMs to generate programs for service mobile robots leveraging mobility, perception, and human interaction skills, and where accurate sequencing and ordering of actions is crucial for success. We contribute CodeBotler, an open-source robot-agnostic tool to program service mobile robots from natural language, and RoboEval, a benchmark for evaluating LLMs' capabilities of generating programs to complete service robot tasks. CodeBotler performs program generation via few-shot prompting of LLMs with an embedded domain-specific language (eDSL) in Python, and leverages skill abstractions to deploy generated programs on any general-purpose mobile robot. RoboEval evaluates the correctness of generated programs by checking execution traces starting with multiple initial states, and checking whether the traces satisfy temporal logic properties that encode correctness for each task. RoboEval also includes multiple prompts per task to test for the robustness of program generation. We evaluate several popular state-of-the-art LLMs with the RoboEval benchmark, and perform a thorough analysis of the modes of failures, resulting in a taxonomy that highlights common pitfalls of LLMs at generating robot programs. We release our code and benchmark at https://amrl.cs.utexas.edu/codebotler/.