mlsanityData to 14 Oct 2026

3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning

arXiv 2411.17735 ↗First posted 23 Nov 2024cs.CV cs.ROcited by 75

Not on the map. Why?

Abstract (arXiv)

Constructing compact and informative 3D scene representations is essential for effective embodied exploration and reasoning, especially in complex environments over extended periods. Existing representations, such as object-centric 3D scene graphs, oversimplify spatial relationships by modeling scenes as isolated objects with restrictive textual relationships, making it difficult to address queries requiring nuanced spatial understanding. Moreover, these representations lack natural mechanisms for active exploration and memory management, hindering their application to lifelong autonomy. In this work, we propose 3D-Mem, a novel 3D scene memory framework for embodied agents. 3D-Mem employs informative multi-view images, termed Memory Snapshots, to represent the scene and capture rich visual information of explored regions. It further integrates frontier-based exploration by introducing Frontier Snapshots-glimpses of unexplored areas-enabling agents to make informed decisions by considering both known and potential new information. To support lifelong memory in active exploration settings, we present an incremental construction pipeline for 3D-Mem, as well as a memory retrieval technique for memory management. Experimental results on three benchmarks demonstrate that 3D-Mem significantly enhances agents' exploration and reasoning capabilities in 3D environments, highlighting its potential for advancing applications in embodied AI.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 39 of its 44 references that are in the map. First posted on arXiv 23 Nov 2024.

Builds on

39 of 44 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  5. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  6. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  7. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  8. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  9. 2023Segment Everything Everywhere All at Oncecited by 548
  10. 2023LERF: Language Embedded Radiance Fieldscited by 543
  11. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  12. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
27 more
  1. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  2. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  3. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  4. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  5. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  6. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  7. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 205
  8. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 178
  9. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  10. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 177
  11. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  12. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
  13. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  14. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  15. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 94
  16. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  17. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 89
  18. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  19. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  20. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representationcited by 54
  21. 2022Topological Semantic Graph Memory for Image-Goal Navigationcited by 45
  22. 2023Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understandingcited by 38
  23. 2022Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understandingcited by 37
  24. 2018Visual Graphs from Motion (VGfM): Scene understanding with object geometry reasoningcited by 27
  25. 2024RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigationcited by 25
  26. 2023VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representationscited by 23
  27. 20223D Concept Grounding on Neural Fieldscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 49.

75 papers.

Sort:

By area:

  1. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 124
  2. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 51
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  4. 2025Learning 3D Persistent Embodied World Modelscited by 29
  5. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  6. 2025Describe Anything Anywhere At Any Momentcited by 19
  7. 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systemscited by 15
  8. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  9. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
  10. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 12
  11. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 11
  12. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
38 more
  1. 2025When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answeringcited by 10
  2. 2025Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoningcited by 9
  3. 2025Toward Ambulatory Vision: Learning Visually-Grounded Active View Selectioncited by 6
  4. 2026NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulationcited by 6
  5. 2025Vision-Language Memory for Spatial Reasoningcited by 5
  6. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5
  7. 2025CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoningcited by 5
  8. 2026GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoningcited by 5
  9. 2025PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agentscited by 4
  10. 2025Expand Your SCOPE: Semantic Cognition over Potential-Based Exploration for Embodied Visual Navigationcited by 4
  11. 2025ReEXplore: Improving MLLMs for Embodied Exploration with Contextualized Retrospective Experience Replaycited by 4
  12. 2026MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planningcited by 4
  13. 2025GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answeringcited by 3
  14. 2025Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulationcited by 3
  15. 2026Bridging the Indoor-Outdoor Gap: Vision-Centric Instruction-Guided Embodied Navigation for the Last Meterscited by 3
  16. 2026HIMM: Human-Inspired Long-Term Memory Modeling for Embodied Exploration and Question Answeringcited by 3
  17. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  18. 2026Planning with the Viewscited by 3
  19. 2026Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guaranteescited by 3
  20. 2025Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answeringcited by 2
  21. 2025R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Spacecited by 2
  22. 2026OnlineSI: Taming Large Language Model for Online 3D Understanding and Groundingcited by 2
  23. 2026LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigationcited by 2
  24. 2026What if? Emulative Simulation with World Models for Situated Reasoningcited by 2
  25. 2026BrainMem: Brain-Inspired Evolving Memory for Embodied Agent Task Planningcited by 2
  26. 2026RAVEN: Long-Horizon Reasoning & Navigation with a Visuo-Spatio-Temporal Memorycited by 2
  27. 2025General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Actingcited by 1
  28. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  29. 2025PIGEON: VLM-Driven Object Navigation via Points of Interest Selectioncited by 1
  30. 2026RenderMem: Rendering as Spatial Memory Retrievalcited by 1
  31. 2026AERR-Nav: Adaptive Exploration-Recovery-Reminiscing Strategy for Zero-Shot Object Navigationcited by 1
  32. 2026EvoMemNav: Efficient Self-Evolving Fine-Grained Memory for Zero-Shot Embodied Navigationcited by 1
  33. 2026CoCoSI: Collaborative Cognitive Map Construction for Spatial Intelligencecited by 1
  34. 2026ObsGraph: Hierarchical Observation Representation for Embodied Reasoning and Explorationcited by 1
  35. 2026RoboAtlas: Contextual Active SLAMcited by 1
  36. 2026Mosaic: Runtime-Efficient Multi-Agent Embodied Planningcited by 1
  37. 2026Self-Evolving Just-In-Time Memory for Proactive Embodied Safetycited by 1
  38. 2026LT-Mem: Volatility-Aware Spatio-Temporal Memory for Lifelong Scene Understandingcited by 1

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agents
  2. 2026CGFM-Nav: Cognitive Graph-Field Memory for Semantic-Guided Lifelong Multimodal Embodied Navigation
  3. 2026EmbodiedMemory-Bench: Benchmarking Embodied Memory for Long-Horizon Embodied Tasks
  4. 2026Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning
  5. 20263D-Belief: Embodied Belief Inference via Generative 3D World Modeling
  6. 2025REM: Evaluating LLM Embodied Spatial Reasoning through Multi-Frame Trajectories
  7. 2025Mem4Nav: Boosting Vision-and-Language Navigation in Urban Environments with a Hierarchical Spatial-Cognition Long-Short Memory System
  8. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding