mlsanityData to 7 Oct 2026

Probing the 3D Awareness of Visual Foundation Models

arXiv 2404.08636 ↗First posted 12 Apr 2024cs.CVcited by 175

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in large-scale pretraining have yielded visual foundation models with strong capabilities. Not only can recent models generalize to arbitrary images for their training task, their intermediate representations are useful for other visual tasks such as detection and segmentation. Given that such models can classify, delineate, and localize objects in 2D, we ask whether they also represent their 3D structure? In this work, we analyze the 3D awareness of visual foundation models. We posit that 3D awareness implies that representations (1) encode the 3D structure of the scene and (2) consistently represent the surface across views. We conduct a series of experiments using task-specific probes and zero-shot inference procedures on frozen features. Our experiments reveal several limitations of the current models. Our code and analysis can be found at https://github.com/mbanani/probe3d.

Key facts. Cited by 175 papers in the map (data through 7 Oct 2026). Builds on 75 of its 102 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

75 of 102 references.

References · 75

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023Segment Anythingcited by 8,158
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  12. 2022A ConvNet for the 2020scited by 3,545
38 more
  1. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  4. 2021Vision Transformers for Dense Predictioncited by 1,789
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  9. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  10. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  11. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  12. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  13. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  14. 2023Vision Transformers Need Registerscited by 844
  15. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  16. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  17. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  18. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  19. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  20. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  21. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  22. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  23. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  24. 2018Do Better ImageNet Models Transfer Better?cited by 560
  25. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  26. 2023Emergent Correspondence from Image Diffusioncited by 437
  27. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  28. 2022DeiT III: Revenge of the ViTcited by 417
  29. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  30. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  31. 2023Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Imagecited by 357
  32. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  33. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  34. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  35. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  36. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  37. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 211
  38. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 53; 2025: 56; 2026: 66.

175 papers.

Sort:

By area:

  1. 2024Depth Anything V2cited by 1,385
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  4. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
  5. 2025FoundationStereo: Zero-Shot Stereo Matchingcited by 168
  6. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  7. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  8. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  9. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 88
  10. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 70
  11. 2024SINGAPO: Single Image Controlled Generation of Articulated Parts in Objectscited by 66
  12. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
38 more
  1. 2024SpotlessSplats: Ignoring Distractors in 3D Gaussian Splattingcited by 61
  2. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 56
  3. 2025DEFOM-Stereo: Depth Foundation Model Based Stereo Matchingcited by 51
  4. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 46
  5. 2024TIPS: Text-Image Pretraining with Spatial awarenesscited by 37
  6. 2025SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Modelscited by 37
  7. 2024GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generationcited by 35
  8. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  9. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  10. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  11. 2024Cross-View Completion Models are Zero-shot Correspondence Estimatorscited by 30
  12. 2024Scaling 4D Representationscited by 30
  13. 2024SPA: 3D Spatial-Awareness Enables Effective Embodied Representationcited by 28
  14. 2024Feat2GS: Probing Visual Foundation Models with Gaussian Splattingcited by 26
  15. 2025AnyUp: Universal Feature Upsamplingcited by 26
  16. 2024ImageNet3D: Towards General-Purpose Object-Level 3D Understandingcited by 25
  17. 2025FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placementcited by 24
  18. 2025LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Modelscited by 23
  19. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  20. 2024DiffSim: Taming Diffusion Models for Evaluating Visual Similaritycited by 20
  21. 2024DepthCues: Evaluating Monocular Depth Perception in Large Vision Modelscited by 18
  22. 2025C3G: Learning Compact 3D Representations with 2K Gaussianscited by 18
  23. 2024PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillationcited by 17
  24. 2024The 3D-PC: a benchmark for visual perspective taking in humans and machinescited by 16
  25. 2024LangOcc: Self-Supervised Open Vocabulary Occupancy Estimation via Volume Renderingcited by 16
  26. 2024Near, far: Patch-ordering enhances vision foundation models' scene understandingcited by 15
  27. 2024Can Visual Foundation Models Achieve Long-term Point Tracking?cited by 14
  28. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  29. 2024Towards Foundation Models for 3D Vision: How Close Are We?cited by 13
  30. 2024Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuningcited by 13
  31. 2025Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understandingcited by 13
  32. 2025Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Modelscited by 13
  33. 2024Duoduo CLIP: Efficient 3D Understanding with Multi-View Imagescited by 12
  34. 2024Evaluating Multiview Object Consistency in Humans and Image Modelscited by 12
  35. 2024SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimationcited by 12
  36. 2025CTRL-O: Language-Controllable Object-Centric Visual Representation Learningcited by 12
  37. 2026Repurposing Geometric Foundation Models for Multi-view Diffusioncited by 12
  38. 2025Compass Control: Multi Object Orientation Control for Text-to-Image Generationcited by 11

50 most cited of 175 All 175 (JSON)