# Advancing 3D Scene Understanding with MV-ScanQA Multi-View Reasoning Evaluation and TripAlign Pre-training Dataset

arXiv: https://arxiv.org/abs/2508.11058 · HTML: https://mlsanity.com/paper/2508.11058/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching](https://mlsanity.com/paper/2311.11284/) (2023)
- [GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models](https://mlsanity.com/paper/2310.08529/) (2023)
- [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020)
- [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023)
- [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023)
- [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023)
- [LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark](https://mlsanity.com/paper/2306.06687/) (2023)
- [Towards Learning a Generalist Model for Embodied Navigation](https://mlsanity.com/paper/2312.02010/) (2023)
- [Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning](https://mlsanity.com/paper/2309.02999/) (2023)
- [Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding](https://mlsanity.com/paper/2305.10714/) (2023)
- [Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA](https://mlsanity.com/paper/2402.15933/) (2024)
- [3D Vision and Language Pretraining with Large-Scale Synthetic Data](https://mlsanity.com/paper/2407.06084/) (2024)
- [Learn 3D VQA Better with Active Selection and Reannotation](https://mlsanity.com/paper/2507.04630/) (2025)

## Built on by (most cited first, 8 of 8)

- [OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding](https://mlsanity.com/paper/2604.25276/) (2026)
- [CrossView Suite: Harnessing Cross-view Spatial Intelligence of MLLMs with Dataset, Model and Benchmark](https://mlsanity.com/paper/2605.18621/) (2026)
- [Temporal-Aware Reasoning Optimization for Video Temporal Grounding](https://mlsanity.com/paper/2606.09248/) (2026)
- [Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs](https://mlsanity.com/paper/2606.01215/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA](https://mlsanity.com/paper/2607.28442/) (2026)
- [ID-VTG: Image-Disambiguated Video Temporal Grounding](https://mlsanity.com/paper/2608.20127/) (2026)
- [SparseTalk - Sparsifying 3D Gaussian Language Fields for Efficient 3D Visual Question Answering](https://mlsanity.com/paper/2609.15137/) (2026)

## Abstract (arXiv)

The advancement of 3D vision-language (3D VL) learning is hindered by several limitations in existing 3D VL datasets: they rarely necessitate reasoning beyond a close range of objects in single viewpoint, and annotations often link instructions to single objects, missing richer contextual alignments between multiple objects. This significantly curtails the development of models capable of deep, multi-view 3D scene understanding over distant objects. To address these challenges, we introduce MV-ScanQA, a novel 3D question answering dataset where 68% of questions explicitly require integrating information from multiple views (compared to less than 7% in existing datasets), thereby rigorously testing multi-view compositional reasoning. To facilitate the training of models for such demanding scenarios, we present TripAlign dataset, a large-scale and low-cost 2D-3D-language pre-training corpus containing 1M <2D view, set of 3D objects, text> triplets that explicitly aligns groups of contextually related objects with text, providing richer, view-grounded multi-object multimodal alignment signals than previous single-object annotations. We further develop LEGO, a baseline method for the multi-view reasoning challenge in MV-ScanQA, transferring knowledge from pre-trained 2D LVLMs to 3D domain with TripAlign. Empirically, LEGO pre-trained on TripAlign achieves state-of-the-art performance not only on the proposed MV-ScanQA, but also on existing benchmarks for 3D dense captioning and question answering. Datasets and code are available at https://matthewdm0816.github.io/tripalign-mvscanqa.