mlsanityData to 14 Oct 2026

Grounding Multimodal Large Language Models in Actions

arXiv 2406.07904 ↗First posted 12 Jun 2024cs.LGcited by 35

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM into different embodiments and their associated action spaces, with the goal of leveraging the multimodal world knowledge of the MLLM. We first generalize a number of methods through a unified architecture and the lens of action space adaptors. For continuous actions, we show that a learned tokenization allows for sufficient modeling precision, yielding the best performance on downstream tasks. For discrete actions, we demonstrate that semantically aligning these actions with the native output token space of the MLLM leads to the strongest performance. We arrive at these lessons via a thorough study of seven action space adapters on five different environments, encompassing over 114 embodied tasks.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 69 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

58 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Vision-Language Foundation Models as Effective Robot Imitators2 Nov
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  3. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 55

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
38 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  10. 2022A Generalist Agentcited by 808
  11. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  12. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  13. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  14. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  15. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  16. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  17. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  18. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  19. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  20. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  21. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  22. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  23. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  24. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 282
  25. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  26. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  27. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  28. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  29. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  30. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  31. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  32. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
  33. 2024Behavior Generation with Latent Actionscited by 160
  34. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  35. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  36. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  37. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  38. 2023Jointly Training Large Autoregressive Multimodal Modelscited by 32

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 16; 2026: 9.

35 papers.

Sort:

By area:

  1. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  2. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 73
  3. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  4. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  5. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 35
  6. 2025BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learningcited by 27
  7. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  8. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  9. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 18
  10. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  11. 2025Guiding Audio Editing with Audio Language Modelcited by 12
  12. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9
23 more
  1. 2024Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervisioncited by 9
  2. 2024QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learningcited by 9
  3. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  4. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 9
  5. 2025MAPS: Preserving Vision-Language Representations via Module-Wise Proximity Scheduling for Better Vision-Language-Action Generalizationcited by 7
  6. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 7
  7. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5
  8. 2025PhysiAgent: An Embodied Agent Framework in Physical Worldcited by 5
  9. 2025Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agentscited by 4
  10. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  11. 2026From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Modelscited by 4
  12. 2026Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agentscited by 3
  13. 2024TSPE: Task-Specific Prompt Ensemble for Improved Zero-Shot Audio Classificationcited by 2
  14. 2026Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasetscited by 2
  15. 2024The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Datacited by 1
  16. 2025Boosting Robotic Manipulation Generalization with Minimal Costly Datacited by 1
  17. 2025Embodied AI in Machine Learning -- is it Really Embodied?cited by 1
  18. 2025From Building Blocks to Planning: Multi-Step Spatial Reasoning in LLMs with Reinforcement Learningcited by 1
  19. 2026AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretationcited by 1
  20. 2025Grounding Foundational Vision Models with 3D Human Poses for Robust Action Recognitioncited by 0
  21. 2026Environmental Understanding Vision-Language Model for Embodied Agentcited by 0
  22. 2026NavOne: One-Step Global Planning for Vision-Language Navigation on Top-Down Mapscited by 0
  23. 2026Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
  2. 2025Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
  3. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
  4. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain
  5. 2025BLM1: A Boundless Large Model for Cross-Space, Cross-Task, and Cross-Embodiment Learning
  6. 2026ST-BiBench: Benchmarking Multi-Stream Multimodal Coordination in Bimanual Embodied Tasks for MLLMs
  7. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks