mlsanityData to 7 Oct 2026

ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation

arXiv 2409.13682 ↗First posted 20 Sep 2024cs.RO cs.AI cs.CLcited by 67

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Navigating and understanding complex environments over extended periods of time is a significant challenge for robots. People interacting with the robot may want to ask questions like where something happened, when it occurred, or how long ago it took place, which would require the robot to reason over a long history of their deployment. To address this problem, we introduce a Retrieval-augmented Memory for Embodied Robots, or ReMEmbR, a system designed for long-horizon video question answering for robot navigation. To evaluate ReMEmbR, we introduce the NaVQA dataset where we annotate spatial, temporal, and descriptive questions to long-horizon robot navigation videos. ReMEmbR employs a structured approach involving a memory building and a querying phase, leveraging temporal information, spatial information, and images to efficiently handle continuously growing robot histories. Our experiments demonstrate that ReMEmbR outperforms LLM and VLM baselines, allowing ReMEmbR to achieve effective long-horizon reasoning with low latency. Additionally, we deploy ReMEmbR on a robot and show that our approach can handle diverse queries. The dataset, code, videos, and other material can be found at the following link: https://nvidia-ai-iot.github.io/remembr

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 47 of its 54 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

47 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Robotics1 Nov
  2. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models26 May
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 44

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
32 more
  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  2. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  5. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  6. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  7. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  8. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  9. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  10. 2017Embodied Question Answeringcited by 335
  11. 2019Vision-and-Dialog Navigationcited by 246
  12. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  13. 2018Semi-parametric Topological Memory for Navigationcited by 203
  14. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  15. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  16. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  17. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  18. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  19. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  20. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  21. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  22. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  23. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  24. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  25. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
  26. 2018Neural Modular Control for Embodied Question Answeringcited by 56
  27. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  28. 2023Deploying and Evaluating LLMs to Program Service Mobile Robotscited by 36
  29. 2022Iterative Vision-and-Language Navigationcited by 34
  30. 2018Shifting the Baseline: Single Modality Performance on Visual Navigation & QAcited by 30
  31. 2024TwoStep: Multi-agent Task Planning using Classical Planners and Large Language Modelscited by 29
  32. 2023Towards Robust Robot 3D Perception in Urban Environments: The UT Campus Object Datasetcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 20; 2026: 43.

67 papers.

Sort:

By area:

  1. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  2. 2026The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Breakcited by 25
  3. 2025Describe Anything Anywhere At Any Momentcited by 19
  4. 2026GAM: Hierarchical Graph-based Agentic Memory for LLM Agentscited by 19
  5. 2026From Storage to Steering: Memory Control Flow Attacks on LLM Agentscited by 17
  6. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  7. 2025LLM-Empowered Embodied Agent for Memory-Augmented Task Planning in Household Roboticscited by 15
  8. 2025CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoningcited by 13
  9. 2026Gated Memory Policy: In-Context Memorization and Adaptationcited by 13
  10. 2025Memory-Centric Embodied Question Answeringcited by 12
  11. 2025Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answeringcited by 12
  12. 2025Episodic memory in AI agents poses risks that should be studied and mitigatedcited by 11
38 more
  1. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agentscited by 11
  2. 2025SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigationcited by 10
  3. 2025Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoningcited by 9
  4. 2026Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflectioncited by 9
  5. 2026MANSION: Multi-floor lANguage-to-3D Scene generatIOn for loNg-horizon taskscited by 8
  6. 2025Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selectioncited by 7
  7. 2024NL-Eye: Abductive NLI for Imagescited by 5
  8. 2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
  9. 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Modelcited by 5
  10. 2024Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experiencecited by 4
  11. 2026OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelismcited by 4
  12. 2026Chameleon: Control-Indexed Prospective Memory for Visuomotor Manipulationcited by 4
  13. 20263D Scene Graphs: Open Challenges and Future Directionscited by 4
  14. 2025GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answeringcited by 3
  15. 2026HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answeringcited by 3
  16. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  17. 2026Stop Wandering: Efficient Vision-Language Navigation via Metacognitive Reasoningcited by 3
  18. 2026Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guaranteescited by 3
  19. 2024Spatially Visual Perception for End-to-End Robotic Learningcited by 2
  20. 2025R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Spacecited by 2
  21. 2026KGLAMP: Knowledge Graph-guided Language model for Adaptive Multi-robot Planning and Replanningcited by 2
  22. 2026Agentic Self-Evolutionary Replanning for Embodied Navigationcited by 2
  23. 2026Semantic Flip: Synthetic OOD Generation for Robust Refusal in Embodied Question Answering and Spatial Localizationcited by 2
  24. 2026RAVEN: Long-Horizon Reasoning & Navigation with a Visuo-Spatio-Temporal Memorycited by 2
  25. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  26. 2025Enhancing Reliability in LLM-Integrated Robotic Systems: A Unified Approach to Security and Safetycited by 1
  27. 2025Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answeringcited by 1
  28. 2025MaP-AVR: A Meta-Action Planner for Agents Leveraging Vision Language Models and Retrieval-Augmented Generationcited by 1
  29. 2026Memory-Guided View Refinement for Dynamic Human-in-the-loop EQAcited by 1
  30. 2026All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptationcited by 1
  31. 2026Binary Tracking for Spatial QA and Navigation with Open Vision-Language Modelscited by 1
  32. 2026Analytic Concept-Centric Memory for Agentic Embodied Manipulationcited by 1
  33. 2026Mosaic: Runtime-Efficient Multi-Agent Embodied Planningcited by 1
  34. 2026VoLN: Vision-Only Long-Horizon Navigation---Paradigm, Benchmark, and Methodcited by 1
  35. 2025RobotFleet: An Open-Source Framework for Centralized Multi-Robot Task Planningcited by 0
  36. 2025PIGEON: VLM-Driven Object Navigation via Points of Interest Selectioncited by 0
  37. 2025TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Makingcited by 0
  38. 2026STaR: Scalable Task-Conditioned Retrieval for Long-Horizon Multimodal Robot Memorycited by 0

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation
  2. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation
  3. 2026ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries
  4. 2026SSMG-Nav: Enhancing Lifelong Object Navigation with Semantic Skeleton Memory Graph
  5. 2026LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video
  6. 2025Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents
  7. 2025MemER: Scaling Up Memory for Robot Control via Experience Retrieval
  8. 2026REMem: Reasoning with Episodic Memory in Language Agent