mlsanityData to 14 Oct 2026

Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agents

arXiv 2308.07241 ↗First posted 14 Aug 2023cs.RO cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

Accomplishing household tasks requires to plan step-by-step actions considering the consequences of previous actions. However, the state-of-the-art embodied agents often make mistakes in navigating the environment and interacting with proper objects due to imperfect learning by imitating experts or algorithmic planners without such knowledge. To improve both visual navigation and object interaction, we propose to consider the consequence of taken actions by CAPEAM (Context-Aware Planning and Environment-Aware Memory) that incorporates semantic context (e.g., appropriate objects to interact with) in a sequence of actions, and the changed spatial arrangement and states of interacted objects (e.g., location that the object has been moved to) in inferring the subsequent actions. We empirically show that the agent with the proposed CAPEAM achieves state-of-the-art performance in various metrics using a challenging interactive instruction following benchmark in both seen and unseen environments by large margins (up to +10.70% in unseen env.).

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 34 of its 43 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

34 of 43 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  9. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  10. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  11. 2017Embodied Question Answeringcited by 336
  12. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
22 more
  1. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  2. 2016Value Iteration Networkscited by 160
  3. 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
  4. 2017IQA: Visual Question Answering in Interactive Environmentscited by 155
  5. 2021FILM: Following Instructions in Language with Modular Methodscited by 121
  6. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  7. 2021Visual Room Rearrangementcited by 102
  8. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
  9. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  10. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  11. 2022Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigationcited by 86
  12. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 81
  13. 2017Gated-Attention Architectures for Task-Oriented Language Groundingcited by 72
  14. 2018Neural Modular Control for Embodied Question Answeringcited by 56
  15. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  16. 2017Visual Semantic Planning using Deep Successor Representationscited by 45
  17. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  18. 2020Factorizing Perception and Policy for Interactive Instruction Followingcited by 29
  19. 2021Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Taskscited by 26
  20. 2019Embodied Vision-and-Language Navigation with Dynamic Convolutional Filterscited by 9
  21. 2022Learning to Act with Affordance-Aware Multimodal Neural SLAMcited by 8
  22. 2022Moment-based Adversarial Training for Embodied Language Comprehensioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 9; 2026: 10.

33 papers.

Sort:

By area:

  1. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 57
  2. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 56
  3. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 39
  4. 2024EPO: Hierarchical LLM Agents with Environment Preference Optimizationcited by 28
  5. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 22
  6. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  7. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  8. 2025CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoningcited by 13
  9. 2025Reinforced Reasoning for Embodied Planningcited by 12
  10. 2024Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examplescited by 10
  11. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  12. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Modelscited by 8
21 more
  1. 2024Embodied Instruction Following in Unknown Environmentscited by 8
  2. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  3. 2024Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Followingcited by 7
  4. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 7
  5. 2024DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Controlcited by 4
  6. 2024Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modelingcited by 4
  7. 2023RoboGPT: an intelligent agent of making embodied long-term decisions for daily instruction taskscited by 3
  8. 2025HybridGen: VLM-Guided Hybrid Planning for Scalable Data Generation of Imitation Learningcited by 3
  9. 2026When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Executioncited by 3
  10. 2025Efficient Navigation in Unknown Indoor Environments with Vision-Language Modelscited by 2
  11. 2025Conditional Multi-Stage Failure Recovery for Embodied Agentscited by 1
  12. 2026HARMONI: Multimodal Personalization of Multi-User Human-Robot Interactions with LLMscited by 1
  13. 2026SVLL: Staged Vision-Language Learning for Physically Grounded Embodied Task Planningcited by 1
  14. 2026Evolvable Embodied Agent for Robotic Manipulation via Long Short-Term Reflection and Optimizationcited by 1
  15. 2026ESCAPE: Episodic Spatial Memory and Adaptive Execution Policy for Long-Horizon Mobile Manipulationcited by 1
  16. 2024HAPFI: History-Aware Planning based on Fused Informationcited by 0
  17. 2025Multi-Objective Planning with Contextual Lexicographic Reward Preferencescited by 0
  18. 2026Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoningcited by 0
  19. 2026ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraintscited by 0
  20. 2026GraphThink: Graph-Enhanced LLM Thinking for Long-Horizon Embodied Task Planningcited by 0
  21. 2026AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalizationcited by 0