What Language Does and What the Evidence Supports: A Functional Role Taxonomy and Evidence Audit of Language Grounding in Embodied Agents
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Foundation models place language throughout embodied agents, but its presence does not show what it contributes or how well that contribution is grounded. This survey separates these two questions. We define five non-exclusive functional roles for language: Specification, Embodied Representation, Action Orchestration, Grounding Regulation, and Execution Coupling. For each role, we trace the path from linguistic content to its embodied consumer and identify the observations or interventions that can test the claimed responsibility. Applying this framework to the reviewed literature reveals a recurring gap between functional use and evidential support. Interpretable or revised linguistic intermediates may be incorrect, go unused, or fail to affect later behavior. Even when actions are directly conditioned on language, system-level success does not by itself isolate language's contribution. We therefore evaluate grounding claim by claim, asking whether the reported evidence supports the specific responsibility assigned to language. Using role claims rather than architectures as the unit of comparison allows us to compare modular and end-to-end embodied agents without extending conclusions beyond the reported evidence.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 72 of its 98 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
72 of 98 references.
Closest ancestors · 1
Same direction, newest first
Other references · 71
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
- 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
- 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
- 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
- 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
- 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
- 2022Visual Language Maps for Robot Navigationcited by 362
- 2023An Embodied Generalist Agent in 3D Worldcited by 355
- 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
38 more
- 2023Language to Rewards for Robotic Skill Synthesiscited by 307
- 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
- 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
- 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
- 2020Experience Grounds Languagecited by 241
- 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
- 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
- 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
- 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
- 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
- 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulationcited by 157
- 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
- 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
- 2023Vision-Language Models as Success Detectorscited by 138
- 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
- 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
- 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 118
- 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
- 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 106
- 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
- 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 99
- 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
- 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
- 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
- 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
- 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
- 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
- 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
- 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
- 2025MemER: Scaling Up Memory for Robot Control via Experience Retrievalcited by 64
- 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
- 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 59
- 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraintscited by 58
- 2025VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulationcited by 54
- 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
- 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
- 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
- 2023Towards A Unified Agent with Foundation Modelscited by 46
50 most cited of 71 All 71 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Synthetic Linguistic Agency: How an Embodied Mortal Agent Learns Linguistic Affordances through Consequential Social Experience
- 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data
- 2026From Cognitive Architectures to Language Agents: A Mechanism-Level Review of Lineage, Convergence, and Migration Gaps
- 2024LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments
- 2026Seeing Isn't Believing: Mitigating Belief Inertia via Active Intervention in Embodied Agents
- 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
- 2026Do Vision-Language-Action Models Mean What They Say? On the Role of Faithfulness in Embodied Reasoning
- 2024"What's my model inside of?": Exploring the role of environments for grounded natural language understanding