mlsanityData to 7 Oct 2026

AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents

arXiv 2401.12963 ↗First posted 23 Jan 2024cs.RO cs.AI cs.CLcited by 83

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Foundation models that incorporate language, vision, and more recently actions have revolutionized the ability to harness internet scale data to reason about useful tasks. However, one of the key challenges of training embodied foundation models is the lack of data grounded in the physical world. In this paper, we propose AutoRT, a system that leverages existing foundation models to scale up the deployment of operational robots in completely unseen scenarios with minimal human supervision. AutoRT leverages vision-language models (VLMs) for scene understanding and grounding, and further uses large language models (LLMs) for proposing diverse and novel instructions to be performed by a fleet of robots. Guiding data collection by tapping into the knowledge of foundation models enables AutoRT to effectively reason about autonomy tradeoffs and safety while significantly scaling up data collection for robot learning. We demonstrate AutoRT proposing instructions to over 20 robots across multiple buildings and collecting 77k real robot episodes via both teleoperation and autonomous robot policies. We experimentally show that such "in-the-wild" data collected by AutoRT is significantly more diverse, and that AutoRT's use of LLMs allows for instruction following data collection robots that can align to human preferences.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 39 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

28 of 39 references.

References · 28

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  11. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  12. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
16 more
  1. 2018Universal Sentence Encodercited by 485
  2. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  3. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  4. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  5. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  6. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  7. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  8. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  9. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  10. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  11. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  12. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  13. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  14. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  15. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  16. 2016More than a Million Ways to Be Pushed: A High-Fidelity Experimental Dataset of Planar Pushingcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 33; 2026: 27.

83 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  2. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  4. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  5. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  6. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 72
  7. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  8. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  9. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  10. 2025Fully Autonomous AI Agents Should Not be Developedcited by 43
  11. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  12. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
38 more
  1. 2025Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learningcited by 37
  2. 2024Multi-Task Interactive Robot Fleet Learning with Visual World Modelscited by 33
  3. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  4. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  5. 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real Worldcited by 31
  6. 2025Safety Guardrails for LLM-Enabled Robotscited by 30
  7. 2025Generating Robot Constitutions & Benchmarks for Semantic Safetycited by 28
  8. 2025MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Modelscited by 27
  9. 2024VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Modelscited by 23
  10. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  11. 2024The Shutdown Problem: An AI Engineering Puzzle for Decision Theoristscited by 22
  12. 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interactioncited by 18
  13. 2024So You Think You Can Scale Up Autonomous Robot Data Collection?cited by 17
  14. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  15. 2025Curating Demonstrations using Online Experiencecited by 16
  16. 2024AnyBimanual: Transferring Unimanual Policy for General Bimanual Manipulationcited by 15
  17. 2025OvercookedV2: Rethinking Overcooked for Zero-Shot Coordinationcited by 14
  18. 2025What Matters for Batch Online Reinforcement Learning in Robotics?cited by 14
  19. 2025MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulationcited by 12
  20. 2024VR-GPT: Visual Language Model for Intelligent Virtual Reality Applicationscited by 9
  21. 2024ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activitiescited by 9
  22. 2024Defining and Evaluating Physical Safety for Large Language Modelscited by 9
  23. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 9
  24. 2024A call for embodied AIcited by 7
  25. 2024AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robotscited by 7
  26. 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harnesscited by 7
  27. 2026Towards the Harness of Embodied Agentscited by 7
  28. 2024Efficiently Identifying Watermarked Segments in Mixed-Source Textscited by 6
  29. 2025Agentic Services Computingcited by 6
  30. 2025Scaling Laws in Scientific Discovery with AI and Robot Scientistscited by 5
  31. 2026PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Executioncited by 5
  32. 2025Graphormer-Guided Task Planning: Beyond Static Rules with LLM Safety Perceptioncited by 4
  33. 2025Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulationcited by 4
  34. 2025Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigationcited by 4
  35. 2025Misalignment or misuse? The AGI alignment tradeoffcited by 4
  36. 2025How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptationcited by 4
  37. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  38. 2024Fourier Controller Networks for Real-Time Decision-Making in Embodied Learningcited by 3

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026GigaBrain-0.7: Scaling Embodied Foundation Models to Emergent Capabilities with a Three-System Architecture
  2. 2025Robot-Powered Data Flywheels: Deploying Robots in the Wild for Continual Data Collection and Foundation Model Adaptation
  3. 2025From Grounding to Manipulation: Case Studies of Foundation Model Integration in Embodied Robotic Systems
  4. 2025Foundation Model Driven Robotics: A Comprehensive Review
  5. 2026The Robot Data Factory ↗
  6. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought
  7. 2025SPEAR-1: Scaling Beyond Robot Demonstrations via 3D Understanding
  8. 2025Large Video Planner Enables Generalizable Robot Control