mlsanityData to 7 Oct 2026

TypeFly: Flying Drones with Large Language Model

arXiv 2312.14950 ↗First posted 8 Dec 2023cs.RO cs.AI cs.HCcited by 23

Not on the map. Why?

Abstract (arXiv)

Recent advancements in robot control using large language models (LLMs) have demonstrated significant potential, primarily due to LLMs' capabilities to understand natural language commands and generate executable plans in various languages. However, in real-time and interactive applications involving mobile robots, particularly drones, the sequential token generation process inherent to LLMs introduces substantial latency, i.e. response time, in control plan generation. In this paper, we present a system called ChatFly that tackles this problem using a combination of a novel programming language called MiniSpec and its runtime to reduce the plan generation time and drone response time. That is, instead of asking an LLM to write a program (robotic plan) in the popular but verbose Python, ChatFly gets it to do it in MiniSpec specially designed for token efficiency and stream interpretation. Using a set of challenging drone tasks, we show that design choices made by ChatFly can reduce up to 62% response time and provide a more consistent user experience, enabling responsive and intelligent LLM-based drone control with efficient completion.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 25 of its 55 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

25 of 55 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  12. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
13 more
  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  2. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  3. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  4. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  5. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  6. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  7. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 197
  8. 2023Large Language Models as General Pattern Machinescited by 196
  9. 2023An LLM Compiler for Parallel Function Callingcited by 170
  10. 2019Online Replanning in Belief Space for Partially Observable Task and Motion Problemscited by 60
  11. 2019Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flightcited by 43
  12. 2023AI-assisted coding: Experiments with GPT-4cited by 43
  13. 2023EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edgecited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 7.

23 papers.

Sort:

By area:

  1. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  2. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  3. 2025BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokenscited by 27
  4. 2024Asynchronous LLM Function Callingcited by 18
  5. 2025See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigationcited by 18
  6. 2025GRaD-Nav++: Vision-Language Model Enabled Visual Drone Navigation with Gaussian Radiance Fields and Differentiable Dynamicscited by 14
  7. 2025Real-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoningcited by 10
  8. 2025Next-Generation LLM for UAV: From Natural Language to Autonomous Flightcited by 10
  9. 2025GSCE: A Prompt Framework with Enhanced Reasoning for Reliable LLM-driven Drone Controlcited by 7
  10. 2024Enhancing LLM-based Autonomous Driving Agents to Mitigate Perception Attackscited by 6
  11. 2025A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Searchcited by 5
  12. 2026Large Language Model-Assisted UAV Operations and Communications: A Multifaceted Survey and Tutorialcited by 4
11 more
  1. 2026From Prompt to Physical Actuation: Holistic Threat Modeling of LLM-Enabled Robotic Systemscited by 3
  2. 2024TimelyLLM: Segmented LLM Serving System for Time-sensitive Robotic Applicationscited by 1
  3. 2025Embodied Crowd Countingcited by 1
  4. 2026PINA: Prompt Injection Attack against Navigation Agentscited by 1
  5. 2025Serve Programs, Not Promptscited by 0
  6. 2025PrivLLMSwarm: Privacy-Preserving LLM-Driven UAV Swarms for Secure IoT Surveillancecited by 0
  7. 2025LEO-RobotAgent: A General-purpose Robotic Agent for Language-driven Embodied Operatorcited by 0
  8. 2026SageSched: Efficient LLM Scheduling Confronting Demand Uncertainty and Hybriditycited by 0
  9. 2026PEACE: A Planner-Executor Agent with Constraint Enforcement for UAVscited by 0
  10. 2026RT-SHCUA: Real-Time Self-Hosted Computer-Use Agent for UAV Controlcited by 0
  11. 2026Zero-Shot Mission-Level Evaluation for Aerial MLLM Agentscited by 0