mlsanityData to 7 Oct 2026

From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons

arXiv 2412.08442 ↗First posted 11 Dec 2024cs.LGcited by 42

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

We examine the capability of Multimodal Large Language Models (MLLMs) to tackle diverse domains that extend beyond the traditional language and vision tasks these models are typically trained on. Specifically, our focus lies in areas such as Embodied AI, Games, UI Control, and Planning. To this end, we introduce a process of adapting an MLLM to a Generalist Embodied Agent (GEA). GEA is a single unified model capable of grounding itself across these varied domains through a multi-embodiment action tokenizer. GEA is trained with supervised learning on a large dataset of embodied experiences and with online RL in interactive simulators. We explore the data and algorithmic choices necessary to develop such a model. Our findings reveal the importance of training with cross-domain data and online RL for building generalist agents. The final GEA model achieves strong generalization performance to unseen tasks across diverse benchmarks compared to other generalist models and benchmark-specific approaches.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 79 of its 96 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

79 of 96 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning14 Jun
  2. 2024Grounding Multimodal Large Language Models in Actions12 Jun
  3. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs8 Apr
  4. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  5. 2023Vision-Language Foundation Models as Effective Robot Imitators2 Nov
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  7. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 72

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  9. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  13. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  14. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  15. 2022A Generalist Agentcited by 802
  16. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  17. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  18. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  19. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  20. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  21. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  22. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  23. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  24. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  25. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  26. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  27. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 426
  28. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  29. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  30. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  31. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  32. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  33. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  34. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  35. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  36. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  37. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  38. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 22; 2026: 19.

42 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  3. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  4. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 142
  5. 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 90
  6. 2025RoboArena: Distributed Real-World Evaluation of Generalist Robot Policiescited by 63
  7. 2026MEM: Multi-Scale Embodied Memory for Vision Language Action Modelscited by 55
  8. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  9. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Modelscited by 36
  10. 2025BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learningcited by 25
  11. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  12. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
30 more
  1. 2025AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenancecited by 15
  2. 2025PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understandingcited by 11
  3. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 10
  4. 2025Multimodal LLM-assisted Evolutionary Search for Programmatic Control Policiescited by 8
  5. 2025BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learningcited by 7
  6. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
  7. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 7
  8. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  9. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 6
  10. 2025OmniSAT: Compact Action Token, Faster Auto Regressioncited by 5
  11. 2025MAPS: Preserving Vision-Language Representations via Module-Wise Proximity Scheduling for Better Vision-Language-Action Generalizationcited by 5
  12. 2026Vesta: A Generalist Embodied Reasoning Modelcited by 5
  13. 2025ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Makingcited by 4
  14. 2025OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worldscited by 4
  15. 2026CHORUS: Decentralized Multi-Embodiment Collaboration with One VLA Policycited by 4
  16. 2025Discover, Learn, and Reinforce: Scaling Vision-Language-Action Pretraining with Diverse RL-Generated Trajectoriescited by 3
  17. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learningcited by 2
  18. 2026MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environmentscited by 2
  19. 2025PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?cited by 1
  20. 2026Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoningcited by 1
  21. 2026Cross-Embodiment Transfer via Behavior-Aligned Representationscited by 1
  22. 2025Oh-A-DINO: Understanding and Enhancing Attribute-Level Information in Self-Supervised Object-Centric Representationscited by 0
  23. 2026HaltNav: Reactive Visual Halting over Lightweight Topological Priors for Robust Vision-Language Navigationcited by 0
  24. 2026Environmental Understanding Vision-Language Model for Embodied Agentcited by 0
  25. 2026Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactionscited by 0
  26. 2026Flow Control: Steering Vision-Language-Action Models with Simple Real-Time Inputscited by 0
  27. 2026CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planningcited by 0
  28. 2026StrixAE: An Intelligent Agent for Audio Enhancement under Complex Distortion Coupling in Real-World Scenarioscited by 0
  29. 2026StateMem: Single-State Residual Memory with Adaptive Inference for Vision-Language-Action Policiescited by 0
  30. 2026Agentic Relative Camera Pose Estimation via Learned Ranking and Verificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agents
  2. 2025EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligence
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
  4. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
  5. 2024GenRL: Multimodal-foundation world models for generalization in embodied agents
  6. 2025Evaluating Multimodal Large Language Models with Daily Composite Tasks in Home Environments
  7. 2025On Path to Multimodal Generalist: General-Level and General-Bench
  8. 2025BiTAgent: A Task-Aware Modular Framework for Bidirectional Coupling between Multimodal Large Language Models and World Models