mlsanityData to 7 Oct 2026

Memory Tree Guided Key Frame Querying for Efficient 3D Question Answering

arXiv 2608.18009 ↗First posted 18 Aug 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Answering questions accurately and efficiently in embodied scenarios presents significant challenges due to limited computational and memory resources for Vision Language Model (VLM) inference. Existing methods adopt visual search key frame retrieval method to select critical question-related key frames for VLM input. However, visual search methods are inefficient because they require visual search among thousands of video frames for each individual user query. In this work, we propose a memory tree guided key frame selection paradigm for efficient 3D question answering in embodied scenarios. Our method leverages a compact and reusable 3D scene representation, termed MemTree3D, which supports real-time online construction leveraging camera 6-DoF poses. MemTree3D captures multi-level 3D scene information, enabling a Large Language Model to efficiently query and retrieve question-relevant key frames through our scoring-based frame selection without reprocessing the entire video stream. On OpenEQA, our method improves the LLM-Match of GPT-4o by 17.4%, LLaVA-OneVision-7B by 5.8%, outperforms existing visual search methods. Our code is available at https://github.com/hsiangwei0903/MemTree3D

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 68 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

54 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 20253D Question Answering via only 2D Vision-Language Models28 May
  2. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  3. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep

Other references · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2025Qwen2.5-VL Technical Reportcited by 5,418
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
38 more
  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  4. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  6. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  8. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  9. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  11. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  12. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  13. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  14. 2017Embodied Question Answeringcited by 335
  15. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  16. 2022BoT-SORT: Robust Associations Multi-Pedestrian Trackingcited by 313
  17. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  18. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  19. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  20. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  21. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  22. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  23. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  24. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  25. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 159
  26. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  27. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  28. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 73
  29. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  30. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  31. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 38
  32. 2022(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answeringcited by 29
  33. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  34. 2025CoS: Chain-of-Shot Prompting for Long Video Understandingcited by 27
  35. 2024SceneGPT: A Language Model for 3D Scene Understandingcited by 12
  36. 2025ToSA: Token Merging with Spatial Awarenesscited by 11
  37. 2025Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumblecited by 9
  38. 2024Advancing Household Robotics: Deep Interactive Reinforcement Learning for Efficient Training and Enhanced Performancecited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing Once is Enough? Online Geometry-Aware Token Pruning for 3D Question Answering
  2. 20253D-MoRe: Unified Modal-Contextual Reasoning for Embodied Question Answering
  3. 2026Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation
  4. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering
  5. 2024Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA
  6. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
  7. 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
  8. 2025GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answering