# Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction

arXiv: https://arxiv.org/abs/2508.05294 · HTML: https://mlsanity.com/paper/2508.05294/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 47 of its 66 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 47)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [Foundation Models in Robotics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2312.07843/) (2023)
- [RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking](https://mlsanity.com/paper/2309.01918/) (2023)
- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023)
- [Large Language Models for Robotics: A Survey](https://mlsanity.com/paper/2311.07226/) (2023)
- [RoboCat: A Self-Improving Generalist Agent for Robotic Manipulation](https://mlsanity.com/paper/2306.11706/) (2023)
- [A survey of agent interoperability protocols: Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent-to-Agent Protocol (A2A), and Agent Network Protocol (ANP)](https://mlsanity.com/paper/2505.02279/) (2025)
- [AIOS: LLM Agent Operating System](https://mlsanity.com/paper/2403.16971/) (2024)
- [M-LLM Based Video Frame Selection for Efficient Video Understanding](https://mlsanity.com/paper/2502.19680/) (2025)
- [AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents](https://mlsanity.com/paper/2401.12963/) (2024)
- [Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation](https://mlsanity.com/paper/2303.03480/) (2023)
- [Real-World Robot Applications of Foundation Models: A Review](https://mlsanity.com/paper/2402.05741/) (2024)
- [ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledge](https://mlsanity.com/paper/2505.21906/) (2025)

## Built on by (most cited first, 19 of 19)

- [Spark: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learning](https://mlsanity.com/paper/2601.20209/) (2026)
- [Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Models](https://mlsanity.com/paper/2605.13119/) (2026)
- [Agentic AI Empowered Intent-Based Networking for 6G](https://mlsanity.com/paper/2601.06640/) (2026)
- [VL4Gaze: Unleashing Vision-Language Models for Gaze Following](https://mlsanity.com/paper/2512.20735/) (2025)
- [From Prompt to Physical Actuation: Holistic Threat Modeling of LLM-Enabled Robotic Systems](https://mlsanity.com/paper/2604.27267/) (2026)
- [Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmark](https://mlsanity.com/paper/2510.02356/) (2025)
- [You Don't Need To Stay in The Loop: An Agentic Robotics Loop for Robot-Policy Improvement](https://mlsanity.com/paper/2608.07555/) (2026)
- [Agentic Scene Policies: Unifying Space, Semantics, and Affordances for Robot Action](https://mlsanity.com/paper/2509.19571/) (2025)
- [ROSBag MCP Server: Analyzing Robot Data with LLMs for Agentic Embodied AI Applications](https://mlsanity.com/paper/2511.03497/) (2025)
- [Leveraging Adaptive Group Negotiation for Heterogeneous Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2602.06967/) (2025)
- [A Framework for Low-Latency, LLM-driven Multimodal Interaction on the Pepper Robot](https://mlsanity.com/paper/2603.21013/) (2026)
- [Agentic AI for Self-Driving Laboratories in Soft Matter: Taxonomy, Benchmarks,and Open Challenges](https://mlsanity.com/paper/2601.17920/) (2026)
- [Adversarial Moral Stress Testing of Large Language Models](https://mlsanity.com/paper/2604.01108/) (2026)
- [ReXSonoVQA: A Video QA Benchmark for Procedure-Centric Ultrasound Understanding](https://mlsanity.com/paper/2604.10916/) (2026)
- [Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence](https://mlsanity.com/paper/2608.06756/) (2026)
- [Toward Unified Robot Learning: Bridging Representation, Vision-Language-Action, and World Models](https://mlsanity.com/paper/2609.03927/) (2026)
- [Finder: Agentic Closed-Loop Object Finding for Embodied Grounding](https://mlsanity.com/paper/2609.18058/) (2026)
- [OverForge: Reasoning Through Strategies and Tactics Helps Cooperative Lifelong Adaptation](https://mlsanity.com/paper/2609.39727/) (2026)
- [Words to World: Benchmarking LLMs in Interactive Physical Environments](https://mlsanity.com/paper/2610.15228/) (2026)

## Abstract (arXiv)

Foundation models, including large language models (LLMs) and vision-language models (VLMs), have recently enabled novel approaches to robot autonomy and human-robot interfaces. In parallel, vision-language-action models (VLAs) or large behavior models (LBMs) are increasing the dexterity and capabilities of robotic systems. This survey paper reviews works that advance agentic applications and architectures, including initial efforts with GPT-style interfaces and more complex systems where AI agents function as coordinators, planners, perception actors, or generalist interfaces. Such agentic architectures allow robots to reason over natural language instructions, invoke APIs, plan task sequences, or assist in operations and diagnostics. In addition to peer-reviewed research, due to the fast-evolving nature of the field, we highlight and include community-driven projects, ROS packages, and industrial frameworks that show emerging trends. We propose a taxonomy for classifying model integration approaches and present a comparative analysis of the role that agents play in different solutions in today's literature.