mlsanityData to 14 Oct 2026

3D Question Answering via only 2D Vision-Language Models

arXiv 2505.22143 ↗First posted 28 May 2025cs.CVcited by 10

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large vision-language models (LVLMs) have significantly advanced numerous fields. In this work, we explore how to harness their potential to address 3D scene understanding tasks, using 3D question answering (3D-QA) as a representative example. Due to the limited training data in 3D, we do not train LVLMs but infer in a zero-shot manner. Specifically, we sample 2D views from a 3D point cloud and feed them into 2D models to answer a given question. When the 2D model is chosen, e.g., LLAVA-OV, the quality of sampled views matters the most. We propose cdViews, a novel approach to automatically selecting critical and diverse Views for 3D-QA. cdViews consists of two key components: viewSelector prioritizing critical views based on their potential to provide answer-specific information, and viewNMS enhancing diversity by removing redundant views based on spatial overlap. We evaluate cdViews on the widely-used ScanQA and SQA benchmarks, demonstrating that it achieves state-of-the-art performance in 3D-QA while relying solely on 2D models without fine-tuning. These findings support our belief that 2D LVLMs are currently the most effective alternative (of the resource-intensive 3D LVLMs) for addressing 3D tasks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 36 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

26 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 20243D Vision and Language Pretraining with Large-Scale Synthetic Data8 Jul
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  3. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding18 May

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  8. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  9. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  10. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  11. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  12. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
11 more
  1. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  2. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  3. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 128
  4. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  5. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  6. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  7. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 49
  8. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 33
  9. 2024Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQAcited by 21
  10. 2024Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarkscited by 10
  11. 2024Rationale-guided Prompting for Knowledge-based Visual Question Answeringcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answeringcited by 3
  2. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  3. 2026Predicting Camera Pose from Perspective Descriptions for Spatial Reasoningcited by 1
  4. 2026Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Informationcited by 1
  5. 2026Scalable Multi-Task Low-Rank Model Adaptationcited by 0
  6. 2026Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videoscited by 0
  7. 2026Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportationcited by 0
  8. 2026GReFEM: Multimodal LLMs as Zero-Shot Semantic Assistants for Physics-Guided 3D Mesh Refinementcited by 0
  9. 2026Memory Tree Guided Key Frame Querying for Efficient 3D Question Answeringcited by 0
  10. 2026CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA
  2. 20233D Concept Learning and Reasoning from Multi-View Images
  3. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering
  4. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
  5. 2026Seeing Once is Enough? Online Geometry-Aware Token Pruning for 3D Question Answering
  6. 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
  7. 2026Do 3D Large Language Models Really Understand 3D Spatial Relationships?
  8. 2024Space3D-Bench: Spatial 3D Question Answering Benchmark