mlsanityData to 7 Oct 2026

Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models

arXiv 2310.15127 ↗First posted 23 Oct 2023cs.AI cs.CL cs.LGcited by 37

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Pre-trained and frozen large language models (LLMs) can effectively map simple scene rearrangement instructions to programs over a robot's visuomotor functions through appropriate few-shot example prompting. To parse open-domain natural language and adapt to a user's idiosyncratic procedures, not known during prompt engineering time, fixed prompts fall short. In this paper, we introduce HELPER, an embodied agent equipped with an external memory of language-program pairs that parses free-form human-robot dialogue into action programs through retrieval-augmented LLM prompting: relevant memories are retrieved based on the current dialogue, instruction, correction, or VLM description, and used as in-context prompt examples for LLM querying. The memory is expanded during deployment to include pairs of user's language and action plans, to assist future inferences and personalize them to the user's language and routines. HELPER sets a new state-of-the-art in the TEACh benchmark in both Execution from Dialog History (EDH) and Trajectory from Dialogue (TfD), with a 1.7x improvement over the previous state-of-the-art for TfD. Our models, code, and video results can be found in our project's website: https://helper-agent-llm.github.io.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 53 of its 98 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

53 of 98 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents28 Aug

Other references · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
38 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  2. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  11. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  12. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  13. 2018On Evaluation of Embodied Navigation Agentscited by 526
  14. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  15. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  16. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  17. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  18. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  19. 2022Visual Language Maps for Robot Navigationcited by 362
  20. 2021True Few-Shot Learning with Language Modelscited by 343
  21. 2020Learning to Explore using Active Neural SLAMcited by 308
  22. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  23. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  24. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  25. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  26. 2020iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenescited by 151
  27. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  28. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  29. 2021SOLQ: Segmenting Objects by Learning Queriescited by 68
  30. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 64
  31. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  32. 2023Modular Visual Question Answering via Code Generationcited by 47
  33. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  34. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  35. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  36. 2016Learning Executable Semantic Parsers for Natural Language Understandingcited by 21
  37. 2022Ask4Help: Learning to Leverage an Expert for Embodied Taskscited by 18
  38. 2022DANLI: Deliberative Agent for Following Natural Language Instructionscited by 15

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 19; 2026: 9.

37 papers.

Sort:

By area:

  1. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  2. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  3. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
  4. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  5. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  6. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  7. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 41
  8. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  9. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  10. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  11. 2024EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agentscited by 24
  12. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
25 more
  1. 2024OSCaR: Object State Captioning and State Change Representationcited by 16
  2. 2025Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilizationcited by 15
  3. 2023Towards Responsible Generative AI: A Reference Architecture for Designing Foundation Model based Agentscited by 13
  4. 2025Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderationcited by 12
  5. 2025HiMA-Ecom: Enabling Joint Training of Hierarchical Multi-Agent E-commerce Assistantscited by 9
  6. 2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
  7. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Modelscited by 8
  8. 2024A Taxonomy of Architecture Options for Foundation Model-based Agents: Analysis and Decision Modelcited by 4
  9. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  10. 2025Safety Must Precede the Deployment of Open-Ended AIcited by 3
  11. 2026Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agentscited by 3
  12. 2025City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learningcited by 2
  13. 2025Memory Transfer Planning: LLM-driven Context-Aware Code Adaptation for Robot Manipulationcited by 2
  14. 2025AgentArcEval: An Architecture Evaluation Method for Foundation Model based Agentscited by 2
  15. 2025R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Spacecited by 2
  16. 2026Agentic Self-Evolutionary Replanning for Embodied Navigationcited by 2
  17. 2025Information Seeking for Robust Decision Making under Partial Observabilitycited by 1
  18. 2024Improving Zero-Shot ObjectNav with Generative Communicationcited by 0
  19. 2025Interpretable Robot Control via Structured Behavior Trees and Large Language Modelscited by 0
  20. 2025Lessons from Neuroscience for AI: How integrating Actions, Compositional Structure and Episodic Memory could enable Safe, Interpretable and Human-Like AIcited by 0
  21. 2026Learning to Forget -- Hierarchical Episodic Memory for Lifelong Robot Deploymentcited by 0
  22. 2026Efficient Skill Grounding via Code Refactoring with Small Language Modelscited by 0
  23. 2026Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agentscited by 0
  24. 2026Self-Evolving Just-In-Time Memory for Proactive Embodied Safetycited by 0
  25. 2026Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
  2. 2024Simulating User Agents for Embodied Conversational-AI
  3. 2026Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions
  4. 2026From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?
  5. 2023LLM as A Robotic Brain: Unifying Egocentric Memory and Control
  6. 2025Ask-to-Clarify: Resolving Instruction Ambiguity through Multi-turn Dialogue
  7. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
  8. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks