mlsanityData to 7 Oct 2026

Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

arXiv 2311.16714 ↗First posted 28 Nov 2023cs.CVcited by 65

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

While large language models (LLMs) excel in a simulated world of texts, they struggle to interact with the more realistic world without perceptions of other modalities such as visual or audio signals. Although vision-language models (VLMs) integrate LLM modules (1) aligned with static image features, and (2) may possess prior knowledge of world dynamics (as demonstrated in the text world), they have not been trained in an embodied visual world and thus cannot align with its dynamics. On the other hand, training an embodied agent in a noisy visual world without expert guidance is often challenging and inefficient. In this paper, we train a VLM agent living in a visual world using an LLM agent excelling in a parallel text world. Specifically, we distill LLM's reflection outcomes (improved actions by analyzing mistakes) in a text world's tasks to finetune the VLM on the same tasks of the visual world, resulting in an Embodied Multi-Modal Agent (EMMA) quickly adapting to the visual world dynamics. Such cross-modality imitation learning between the two parallel worlds is achieved by a novel DAgger-DPO algorithm, enabling EMMA to generalize to a broad scope of new tasks without any further guidance from the LLM expert. Extensive evaluations on the ALFWorld benchmark's diverse tasks highlight EMMA's superior performance to SOTA VLM-based agents, e.g., 20%-70% improvement in the success rate.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 53 of its 78 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

53 of 78 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents29 Jan
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  13. 2016Generative Adversarial Imitation Learningcited by 1,576
  14. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  15. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  16. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  17. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  18. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  19. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  20. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  21. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  22. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  23. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  24. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  25. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  26. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  27. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  28. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  29. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  30. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  31. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  32. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  33. 2015Learning to Search Better Than Your Teachercited by 50
  34. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  35. 2016Pointing the Unknown Wordscited by 42
  36. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
  37. 2021Language Models are Few-Shot Butlerscited by 27
  38. 2023Module-wise Adaptive Distillation for Multimodality Foundation Modelscited by 12

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 29; 2026: 22.

65 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  3. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systemscited by 112
  4. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 105
  5. 2024Large Multimodal Agents: A Surveycited by 99
  6. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  7. 2025Memp: Exploring Agent Procedural Memorycited by 88
  8. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  9. 2024AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agentscited by 36
  10. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  11. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  12. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
38 more
  1. 2024WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agentscited by 24
  2. 2024Do We Really Need a Complex Agent System? Distill Embodied Agent into a Single Modelcited by 19
  3. 2024Hazards in Daily Life? Enabling Robots to Proactively Detect and Resolve Anomaliescited by 19
  4. 2024VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-usecited by 17
  5. 2025PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachabilitycited by 17
  6. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  7. 2025Graph World Modelcited by 14
  8. 2026EvoRoute: Experience-Driven Self-Routing LLM Agent Systemscited by 13
  9. 2025Enhancing Multi-Agent Systems via Reinforcement Learning with LLM-based Planner and Graph-based Policycited by 12
  10. 2024SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editingcited by 11
  11. 2025ViC-Bench: Benchmarking Visual-Interleaved Chain-of-Thought Capability in MLLMs with Free-Style Intermediate State Representationscited by 11
  12. 2025Pre3: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generationcited by 11
  13. 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experiencecited by 11
  14. 2024Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examplescited by 9
  15. 2024Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Modelscited by 8
  16. 2024System-2 Mathematical Reasoning via Enriched Instruction Tuningcited by 7
  17. 2025BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learningcited by 7
  18. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 6
  19. 2025WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agentscited by 5
  20. 2025EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLMcited by 5
  21. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5
  22. 2026AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learningcited by 5
  23. 2024KUNPENG: An Embodied Large Model for Intelligent Maritimecited by 4
  24. 2025SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Taskscited by 4
  25. 2026Beyond Experience Retrieval: Learning to Generate Utility-Optimized Structured Experience for Frozen LLMscited by 4
  26. 2025EmbodiedAgent: A Scalable Hierarchical Approach to Overcome Practical Challenge in Multi-Robot Controlcited by 3
  27. 2025Guided Policy Optimization under Partial Observabilitycited by 3
  28. 2025Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensitycited by 3
  29. 2025GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Trainingcited by 3
  30. 2025VL4Gaze: Unleashing Vision-Language Models for Gaze Followingcited by 3
  31. 2026Physical Prompt Injection Attacks on Large Vision-Language Modelscited by 3
  32. 2026Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agentscited by 3
  33. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  34. 2025MCTS-EP: Empowering Embodied Planning with Online Preference Optimizationcited by 2
  35. 2025Human-Centric Open-Future Task Discovery: Formulation, Benchmark, and Scalable Tree-Based Searchcited by 2
  36. 2026StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systemscited by 2
  37. 2026MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environmentscited by 2
  38. 2024Relevance-driven Decision Making for Safer and More Efficient Human Robot Collaborationcited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
  2. 2023Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds
  3. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Models
  4. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
  5. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons
  6. 2025BiTAgent: A Task-Aware Modular Framework for Bidirectional Coupling between Multimodal Large Language Models and World Models
  7. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
  8. 2025BLM1: A Boundless Large Model for Cross-Space, Cross-Task, and Cross-Embodiment Learning