# Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA

arXiv: https://arxiv.org/abs/2402.15933 · HTML: https://mlsanity.com/paper/2402.15933/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 24 Feb 2024.

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021)
- [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023)
- [CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes](https://mlsanity.com/paper/2304.06061/) (2023)
- [VQA-LOL: Visual Question Answering under the Lens of Logic](https://mlsanity.com/paper/2002.08325/) (2020)
- [Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baseline](https://mlsanity.com/paper/2209.12028/) (2022)
- [Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding](https://mlsanity.com/paper/2305.10714/) (2023)
- [Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes](https://mlsanity.com/paper/2306.02329/) (2023)

## Built on by (most cited first, 21 of 21)

- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [Training-free Video Temporal Grounding using Large-scale Pre-trained Models](https://mlsanity.com/paper/2408.16219/) (2024)
- [3D Vision and Language Pretraining with Large-Scale Synthetic Data](https://mlsanity.com/paper/2407.06084/) (2024)
- [RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentation](https://mlsanity.com/paper/2412.02402/) (2024)
- [Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering](https://mlsanity.com/paper/2502.00342/) (2025)
- [3D Question Answering via only 2D Vision-Language Models](https://mlsanity.com/paper/2505.22143/) (2025)
- [CoV: Chain-of-View Prompting for Spatial Reasoning](https://mlsanity.com/paper/2601.05172/) (2026)
- [Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration](https://mlsanity.com/paper/2508.03207/) (2025)
- [Advancing 3D Scene Understanding with MV-ScanQA Multi-View Reasoning Evaluation and TripAlign Pre-training Dataset](https://mlsanity.com/paper/2508.11058/) (2025)
- [ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding](https://mlsanity.com/paper/2408.16314/) (2024)
- [AR-VRM: Imitating Human Motions for Visual Robot Manipulation with Analogical Reasoning](https://mlsanity.com/paper/2508.07626/) (2025)
- [Learn 3D VQA Better with Active Selection and Reannotation](https://mlsanity.com/paper/2507.04630/) (2025)
- [TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring](https://mlsanity.com/paper/2508.04943/) (2025)
- [SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answering](https://mlsanity.com/paper/2603.29437/) (2026)
- [Investigating Domain Gaps for Indoor 3D Object Detection](https://mlsanity.com/paper/2508.17439/) (2025)
- [Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs](https://mlsanity.com/paper/2606.01215/) (2026)
- [FlexMUSE: Multimodal Unification and Semantics Enhancement Framework with Flexible interaction for Creative Writing](https://mlsanity.com/paper/2508.16230/) (2025)
- [Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models](https://mlsanity.com/paper/2603.24721/) (2026)
- [HCNQA: Enhancing 3D VQA with Hierarchical Concentration Narrowing Supervision](https://mlsanity.com/paper/2507.01800/) (2025)
- [SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answering](https://mlsanity.com/paper/2511.14567/) (2025)
- [Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation](https://mlsanity.com/paper/2606.03100/) (2026)

## Abstract (arXiv)

In 3D Visual Question Answering (3D VQA), the scarcity of fully annotated data and limited visual content diversity hampers the generalization to novel scenes and 3D concepts (e.g., only around 800 scenes are utilized in ScanQA and SQA dataset). Current approaches resort supplement 3D reasoning with 2D information. However, these methods face challenges: either they use top-down 2D views that introduce overly complex and sometimes question-irrelevant visual clues, or they rely on globally aggregated scene/image-level representations from 2D VLMs, losing the fine-grained vision-language correlations. To overcome these limitations, our approach utilizes question-conditional 2D view selection procedure, pinpointing semantically relevant 2D inputs for crucial visual clues. We then integrate this 2D knowledge into the 3D-VQA system via a two-branch Transformer structure. This structure, featuring a Twin-Transformer design, compactly combines 2D and 3D modalities and captures fine-grained correlations between modalities, allowing them mutually augmenting each other. Integrating proposed mechanisms above, we present BridgeQA, that offers a fresh perspective on multi-modal transformer-based architectures for 3D-VQA. Experiments validate that BridgeQA achieves state-of-the-art on 3D-VQA datasets and significantly outperforms existing solutions. Code is available at hrefhttps://github.com/matthewdm0816/BridgeQAthis URL.