mlsanityData to 7 Oct 2026

Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation

arXiv 2409.18313 ↗First posted 26 Sep 2024cs.RO cs.AI cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

There is no limit to how much a robot might explore and learn, but all of that knowledge needs to be searchable and actionable. Within language research, retrieval augmented generation (RAG) has become the workhorse of large-scale non-parametric knowledge; however, existing techniques do not directly transfer to the embodied domain, which is multimodal, where data is highly correlated, and perception requires abstraction. To address these challenges, we introduce Embodied-RAG, a framework that enhances the foundational model of an embodied agent with a non-parametric memory system capable of autonomously constructing hierarchical knowledge for both navigation and language generation. Embodied-RAG handles a full range of spatial and semantic resolutions across diverse environments and query types, whether for a specific object or a holistic description of ambiance. At its core, Embodied-RAG's memory is structured as a semantic forest, storing language descriptions at varying levels of detail. This hierarchical organization allows the system to efficiently generate context-sensitive outputs across different robotic platforms. We demonstrate that Embodied-RAG effectively bridges RAG to the robotics domain, successfully handling over 250 explanation and navigation queries across kilometer-level environments, highlighting its promise as a general-purpose non-parametric system for embodied agents.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 43 of its 56 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

43 of 56 references.

References · 43

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  10. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  11. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  12. 2018On Evaluation of Embodied Navigation Agentscited by 526
31 more
  1. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 517
  2. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  3. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  4. 2022Visual Language Maps for Robot Navigationcited by 362
  5. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  6. 2017Embodied Question Answeringcited by 335
  7. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  8. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  9. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  10. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  11. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  12. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  13. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 175
  14. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  15. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  16. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  17. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  18. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  19. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  20. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  21. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
  22. 2023GOAT: GO to Any Thingcited by 89
  23. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  24. 2019Multi-Target Embodied Question Answeringcited by 74
  25. 2021Memory-Augmented Reinforcement Learning for Image-Goal Navigationcited by 64
  26. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  27. 2022Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instancescited by 53
  28. 2021Knowledge-based Embodied Question Answeringcited by 29
  29. 2022Don't Copy the Teacher: Data and Model Challenges in Embodied Dialoguecited by 10
  30. 2023A System for Generalized 3D Multi-Object Searchcited by 9
  31. 2022Self-Supervised Object Goal Navigation with In-Situ Finetuningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 22; 2026: 21.

44 papers.

Sort:

By area:

  1. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  2. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  3. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  4. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  5. 2025RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Explorationcited by 23
  6. 2025NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLMcited by 18
  7. 2025Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilizationcited by 15
  8. 2025Memory-Centric Embodied Question Answeringcited by 12
  9. 2025Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answeringcited by 12
  10. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agentscited by 11
  11. 2025Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Designcited by 11
  12. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
32 more
  1. 2025RAVEN: Resilient Aerial Navigation via Open-Set Semantic Memory and Behavior Adaptationcited by 10
  2. 2025Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoningcited by 9
  3. 2025A Grounded Memory System For Smart Personal Assistantscited by 5
  4. 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Modelcited by 5
  5. 2026Chameleon: Control-Indexed Prospective Memory for Visuomotor Manipulationcited by 4
  6. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  7. 2025GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answeringcited by 3
  8. 2025ELHPlan: Efficient Long-Horizon Task Planning for Multi-Agent Collaborationcited by 3
  9. 2025Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulationcited by 3
  10. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  11. 2026Stop Wandering: Efficient Vision-Language Navigation via Metacognitive Reasoningcited by 3
  12. 2025RANa: Retrieval-Augmented Navigationcited by 2
  13. 2025Multimodal Data Storage and Retrieval for Embodied AI: A Surveycited by 2
  14. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  15. 2026ECHO: Continuous Hierarchical Memory for Vision-Language-Action Modelscited by 2
  16. 2026CLOSER-VLN: Closed-Loop Self-Verified Retrieval-Augmented Reasoning for Aerial Vision-Language Navigationcited by 2
  17. 2025General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Actingcited by 1
  18. 2025MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioningcited by 1
  19. 2025Optimizing Retrieval for RAG via Reinforcement Learningcited by 1
  20. 2026Memento: Towards Proactive Visualization of Everyday Memories with Personal Wearable AR Assistantcited by 1
  21. 2026Memory-Guided View Refinement for Dynamic Human-in-the-loop EQAcited by 1
  22. 2026TrajRAG: Retrieving Geometric-Semantic Experience for Zero-Shot Object Navigationcited by 1
  23. 2026PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraftcited by 1
  24. 2026Binary Tracking for Spatial QA and Navigation with Open Vision-Language Modelscited by 1
  25. 2025Exploring the Link Between Bayesian Inference and Embodied Intelligence: Toward Open Physical-World Embodied AI Systemscited by 0
  26. 2026A Survey of Spatial Memory Representations for Efficient Robot Navigationcited by 0
  27. 2026G-DRAGON: Geospatial Reasoning and Dynamic Planning for Retrieval-Augmented Outdoor Navigationcited by 0
  28. 2026What Language Does and What the Evidence Supports: A Functional Role Taxonomy and Evidence Audit of Language Grounding in Embodied Agentscited by 0
  29. 2026GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenescited by 0
  30. 2026Informative Viewpoint Selection for Episodic-Memory Embodied Question Answering using Omnidirectional Imagescited by 0
  31. 2026Inline Memory Meets Reusable Skills: Memory-centric Framework for Vision-Language-Action Modelcited by 0
  32. 2026EmbodiedRSI: Active Continual Robot Learning Through Hypothesis-Guided Co-Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Task
  2. 2026EmbodiedLGR: Integrating Lightweight Graph Representation and Retrieval for Semantic-Spatial Memory in Robotic Agents
  3. 2026Retrieval-Augmented Robots via Retrieve-Reason-Act
  4. 2026eMEM: A Hybrid Spatio-Temporal Memory System For Embodied Agents
  5. 2023LLM as A Robotic Brain: Unifying Egocentric Memory and Control
  6. 2024Retrieval-Augmented Embodied Agents
  7. 2026Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions
  8. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigation