mlsanityData to 7 Oct 2026

Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding

arXiv 2409.03757 ↗First posted 5 Sep 2024cs.CV cs.AI cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

Complex 3D scene understanding has gained increasing attention, with scene encoding strategies playing a crucial role in this success. However, the optimal scene encoding strategies for various scenarios remain unclear, particularly compared to their image-based counterparts. To address this issue, we present a comprehensive study that probes various visual encoding models for 3D scene understanding, identifying the strengths and limitations of each model across different scenarios. Our evaluation spans seven vision foundation encoders, including image-based, video-based, and 3D foundation models. We evaluate these models in four tasks: Vision-Language Scene Reasoning, Visual Grounding, Segmentation, and Registration, each focusing on different aspects of scene understanding. Our evaluations yield key findings: DINOv2 demonstrates superior performance, video models excel in object-level tasks, diffusion models benefit geometric tasks, and language-pretrained models show unexpected limitations in language-related tasks. These insights challenge some conventional understandings, provide novel perspectives on leveraging visual foundation models, and highlight the need for more flexible encoder selection in future vision-language and scene-understanding tasks. Code: https://github.com/YunzeMan/Lexicon3D

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 99 of its 119 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

99 of 119 references.

References · 99

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2023Segment Anythingcited by 8,158
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  13. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  14. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  15. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  16. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  17. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  18. 2016Colorful Image Colorizationcited by 1,113
  19. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  20. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  21. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  22. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  23. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  24. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  25. 2022Language-driven Semantic Segmentationcited by 711
  26. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  27. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  28. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  29. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  30. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  31. 2023LERF: Language Embedded Radiance Fieldscited by 538
  32. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  33. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  34. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  35. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  36. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  37. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  38. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 25; 2026: 19.

46 papers.

Sort:

By area:

  1. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  2. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  3. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  4. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  5. 2025Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Spacecited by 37
  6. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  7. 2025Embodied AI: From LLMs to World Modelscited by 23
  8. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  9. 2024DepthCues: Evaluating Monocular Depth Perception in Large Vision Modelscited by 18
  10. 2025From Image to Video: An Empirical Study of Diffusion Representationscited by 14
  11. 2025Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understandingcited by 13
  12. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyondcited by 12
34 more
  1. 20253D Question Answering via only 2D Vision-Language Modelscited by 10
  2. 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
  3. 2025SCA3D: Enhancing Cross-modal 3D Retrieval via 3D Shape and Caption Paired Data Augmentationcited by 9
  4. 2025Bridging Perspectives: Foundation Model Guided BEV Maps for 3D Object Detection and Trackingcited by 9
  5. 2025How Much 3D Do Video Foundation Models Encode?cited by 9
  6. 2025Understanding Multi-View Transformerscited by 8
  7. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 6
  8. 2025EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretrainingcited by 5
  9. 2025The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? A Bias-Controlled Studycited by 5
  10. 2025Masked Scene Modeling: Narrowing the Gap Between Supervised and Self-Supervised Learning in 3D Scene Understandingcited by 4
  11. 2025Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perceptioncited by 4
  12. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
  13. 2025View-Consistent Diffusion Representations for 3D-Consistent Video Generationcited by 3
  14. 20263AM: 3egment Anything with Geometric Consistency in Videoscited by 3
  15. 2025GIQ: Benchmarking 3D Geometric Reasoning of Vision Foundation Models with Simulated and Real Polyhedracited by 2
  16. 2025Diffusion-Based Action Recognition Generalizes to Untrained Domainscited by 2
  17. 2026Capturing Visual Environment Structure Correlates with Control Performancecited by 2
  18. 2026DINOLight: Robust Ambient Light Normalization with Self-supervised Visual Prior Integrationcited by 2
  19. 2026SpatialBoost: Enhancing Visual Representation through Language-Guided Reasoningcited by 2
  20. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1
  21. 20263D-IDE: 3D Implicit Depth Emergentcited by 1
  22. 2026The Variance Brain Foundation Models Forgot: Third-Order Statistics Predict Cognition Where Billion-Parameter Models Failcited by 1
  23. 2026SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Videocited by 1
  24. 2026SeeSE3: Emergence of 3D Space in Vision Featurescited by 1
  25. 2026On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-trainingcited by 0
  26. 2026SHED Light on Segmentation for Dense Predictioncited by 0
  27. 2026Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurementcited by 0
  28. 2026Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understandingcited by 0
  29. 2026Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Taskcited by 0
  30. 2026The JEPA Predictor: A Transferable Operator for Occluded Feature Completioncited by 0
  31. 2026From Recovery to Drop-off: How Action Post-training Reduces a VLM's Late-Layer Depth Decodabilitycited by 0
  32. 2026DINOcular: Self-Supervised Visuospatial Representationscited by 0
  33. 2026What, Where, and How: Probing Spatiotemporal Representations in Video Foundation Modelscited by 0
  34. 2026NeuSOGA3D: A Neuro-Symbolic Framework for Explainable 3D Geometric Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
  2. 2026Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?
  3. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
  4. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding
  5. 2025Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models
  6. 2025Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding
  7. 2025Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding
  8. 2025Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding