mlsanityData to 14 Oct 2026

3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment

arXiv 2308.04352 ↗First posted 8 Aug 2023cs.CVcited by 204

Not on the map. Why?

Cited by 69 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

3D vision-language grounding (3D-VL) is an emerging field that aims to connect the 3D physical world with natural language, which is crucial for achieving embodied intelligence. Current 3D-VL models rely heavily on sophisticated modules, auxiliary losses, and optimization tricks, which calls for a simple and unified model. In this paper, we propose 3D-VisTA, a pre-trained Transformer for 3D Vision and Text Alignment that can be easily adapted to various downstream tasks. 3D-VisTA simply utilizes self-attention layers for both single-modal modeling and multi-modal fusion without any sophisticated task-specific design. To further enhance its performance on 3D-VL tasks, we construct ScanScribe, the first large-scale 3D scene-text pairs dataset for 3D-VL pre-training. ScanScribe contains 2,995 RGB-D scans for 1,185 unique indoor scenes originating from ScanNet and 3R-Scan datasets, along with paired 278K scene descriptions generated from existing 3D-VL tasks, templates, and GPT-3. 3D-VisTA is pre-trained on ScanScribe via masked language/object modeling and scene-text matching. It achieves state-of-the-art results on various 3D-VL tasks, ranging from visual grounding and dense captioning to question answering and situated reasoning. Moreover, 3D-VisTA demonstrates superior data efficiency, obtaining strong performance even with limited annotations during downstream task fine-tuning.

Key facts. Cited by 204 papers in the map (data through 14 Oct 2026). Builds on 45 of its 60 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

45 of 60 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
33 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  5. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  9. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  11. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  12. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  13. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  14. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  15. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  16. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  17. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  18. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  19. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
  20. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  21. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
  22. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  23. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  24. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  25. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  26. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 110
  27. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  28. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 89
  29. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  30. 2019VoteNet: A Deep Learning Label Fusion Method for Multi-Atlas Segmentationcited by 44
  31. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  32. 2022Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baselinecited by 25
  33. 2021VLGrammar: Grounded Grammar Induction of Vision and Languagecited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 50; 2025: 93; 2026: 51.

204 papers.

Sort:

By area:

  1. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  2. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  3. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  4. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  5. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  6. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  7. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  8. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  9. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
  10. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  11. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  12. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
38 more
  1. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  2. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 118
  3. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  4. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  5. 2025Visual Spatial Tuningcited by 89
  6. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 80
  7. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  8. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  9. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  10. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 68
  11. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
  12. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 63
  13. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  14. 2024Multi-modal Situated Reasoning in 3D Scenescited by 51
  15. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  16. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 51
  17. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 49
  18. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  19. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  20. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  21. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  22. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  23. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 35
  24. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 33
  25. 20253D Aware Region Prompted Vision Language Modelcited by 33
  26. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  27. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 27
  28. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 27
  29. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  30. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
  31. 2024RefMask3D: Language-Guided Transformer for 3D Referring Segmentationcited by 25
  32. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  33. 2025SplatTalk: 3D VQA with Gaussian Splattingcited by 24
  34. 2025OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understandingcited by 24
  35. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  36. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  37. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 22
  38. 2024Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQAcited by 21

50 most cited of 204 All 204 (JSON)