mlsanityData to 7 Oct 2026

OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Following

arXiv 2403.03017 ↗First posted 5 Mar 2024cs.AIcited by 14

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Embodied Instruction Following (EIF) is a crucial task in embodied learning, requiring agents to interact with their environment through egocentric observations to fulfill natural language instructions. Recent advancements have seen a surge in employing large language models (LLMs) within a framework-centric approach to enhance performance in embodied learning tasks, including EIF. Despite these efforts, there exists a lack of a unified understanding regarding the impact of various components-ranging from visual perception to action execution-on task performance. To address this gap, we introduce OPEx, a comprehensive framework that delineates the core components essential for solving embodied learning tasks: Observer, Planner, and Executor. Through extensive evaluations, we provide a deep analysis of how each component influences EIF task performance. Furthermore, we innovate within this space by deploying a multi-agent dialogue strategy on a TextWorld counterpart, further enhancing task performance. Our findings reveal that LLM-centric design markedly improves EIF outcomes, identify visual perception and low-level action execution as critical bottlenecks, and demonstrate that augmenting LLMs with a multi-agent framework further elevates performance.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 35 of its 51 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

35 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 34

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2017Mask R-CNNcited by 7,259
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  12. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
22 more
  1. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  2. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  3. 2023Language Models can Solve Computer Taskscited by 371
  4. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  5. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  6. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  7. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planningcited by 204
  8. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  9. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  10. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  11. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  12. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  13. 2021Skill Induction and Planning with Latent Languagecited by 99
  14. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  15. 2023CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalizationcited by 88
  16. 2021SOLQ: Segmenting Objects by Learning Queriescited by 68
  17. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  18. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  19. 2020Factorizing Perception and Policy for Interactive Instruction Followingcited by 28
  20. 2021Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Taskscited by 25
  21. 2022LEBP -- Language Expectation & Binding Policy: A Two-Stream Framework for Embodied Vision-and-Language Interaction Task Learning Agentscited by 11
  22. 2021Modular Framework for Visuomotor Language Groundingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 4.

14 papers.

Sort:

By area:

  1. 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learningcited by 74
  2. 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planningcited by 37
  3. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spacescited by 33
  4. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
  5. 2024Enhancing Agent Learning through World Dynamics Modelingcited by 9
  6. 2025Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learningcited by 8
  7. 2026How to Interpret Agent Behaviorcited by 8
  8. 2025gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttlingcited by 4
  9. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  10. 2025ELHPlan: Efficient Long-Horizon Task Planning for Multi-Agent Collaborationcited by 3
  11. 2026When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Makingcited by 2
  12. 2025OceanGym: A Benchmark Environment for Underwater Embodied Agentscited by 0
2 more
  1. 2026Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Modelscited by 0
  2. 2026GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Models
  2. 2026How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective
  3. 2024LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments
  4. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
  5. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making
  6. 2026RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
  7. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
  8. 2025Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis