# Feat2GS: Probing Visual Foundation Models with Gaussian Splatting

arXiv: https://arxiv.org/abs/2412.09606 · HTML: https://mlsanity.com/paper/2412.09606/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 105 of its 125 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Builds on (most cited first, 30 of 105)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)

## Built on by (most cited first, 27 of 27)

- [TTT3R: 3D Reconstruction as Test-Time Training](https://mlsanity.com/paper/2509.26645/) (2025)
- [Easi3R: Estimating Disentangled Motion from DUSt3R Without Training](https://mlsanity.com/paper/2503.24391/) (2025)
- [Human3R: Everyone Everywhere All at Once](https://mlsanity.com/paper/2510.06219/) (2025)
- [DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers](https://mlsanity.com/paper/2503.14405/) (2025)
- [VGGT4D: Mining Motion Cues in Visual Geometry Transformers for 4D Scene Reconstruction](https://mlsanity.com/paper/2511.19971/) (2025)
- [Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning](https://mlsanity.com/paper/2507.14137/) (2025)
- [SemanticSplat: Feed-Forward 3D Scene Understanding with Language-Aware Gaussian Fields](https://mlsanity.com/paper/2506.09565/) (2025)
- [CF3: Compact and Fast 3D Feature Fields](https://mlsanity.com/paper/2508.05254/) (2025)
- [How Much 3D Do Video Foundation Models Encode?](https://mlsanity.com/paper/2512.19949/) (2025)
- [UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding](https://mlsanity.com/paper/2508.11952/) (2025)
- [GECO: Geometrically Consistent Embedding with Lightspeed Inference](https://mlsanity.com/paper/2508.00746/) (2025)
- [Fin3R: Fine-tuning Feed-forward 3D Reconstruction Models via Monocular Knowledge Distillation](https://mlsanity.com/paper/2511.22429/) (2025)
- [Selfi: Self Improving Reconstruction Engine via 3D Geometric Feature Alignment](https://mlsanity.com/paper/2512.08930/) (2025)
- [Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models](https://mlsanity.com/paper/2507.00493/) (2025)
- [The Less You Depend, The More You Learn: Synthesizing Novel Views from Sparse, Unposed Images with Minimal 3D Knowledge](https://mlsanity.com/paper/2506.09885/) (2025)
- [Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation](https://mlsanity.com/paper/2505.02075/) (2025)
- [XYZCylinder: Towards Compatible Feed-Forward 3D Gaussian Splatting for Driving Scenes via Unified Cylinder Lifting Method](https://mlsanity.com/paper/2510.07856/) (2025)
- [On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training](https://mlsanity.com/paper/2601.18929/) (2026)
- [Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning](https://mlsanity.com/paper/2601.22231/) (2026)
- [Spectral Probing of Feature Upsamplers in 2D-to-3D Scene Reconstruction](https://mlsanity.com/paper/2603.05787/) (2026)
- [Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement](https://mlsanity.com/paper/2603.06459/) (2026)
- [PhyGaP: Physically-Grounded Gaussians with Polarization Cues](https://mlsanity.com/paper/2603.14001/) (2026)
- [GaussiAnimate: Rig Animatable Categories with Level of Dynamics](https://mlsanity.com/paper/2604.08547/) (2026)
- [SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision](https://mlsanity.com/paper/2606.19915/) (2026)
- [SUFLECA: Scaling Up Feature Learning for CAD-to-image Alignment](https://mlsanity.com/paper/2607.15058/) (2026)
- [D3GS: Depth, DINO, and RGB Diffusion Co-Guided 3D Gaussian Splatting for Sparse-View Reconstruction](https://mlsanity.com/paper/2609.22941/) (2026)
- [Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis](https://mlsanity.com/paper/2610.03717/) (2026)

## Abstract (arXiv)

Given that visual foundation models (VFMs) are trained on extensive datasets but often limited to 2D images, a natural question arises: how well do they understand the 3D world? With the differences in architecture and training protocols (i.e., objectives, proxy tasks), a unified framework to fairly and comprehensively probe their 3D awareness is urgently needed. Existing works on 3D probing suggest single-view 2.5D estimation (e.g., depth and normal) or two-view sparse 2D correspondence (e.g., matching and tracking). Unfortunately, these tasks ignore texture awareness, and require 3D data as ground-truth, which limits the scale and diversity of their evaluation set. To address these issues, we introduce Feat2GS, which readout 3D Gaussians attributes from VFM features extracted from unposed images. This allows us to probe 3D awareness for geometry and texture via novel view synthesis, without requiring 3D data. Additionally, the disentanglement of 3DGS parameters - geometry (boldsymbolx, α, Sigma) and texture (boldsymbolc) - enables separate analysis of texture and geometry awareness. Under Feat2GS, we conduct extensive experiments to probe the 3D awareness of several VFMs, and investigate the ingredients that lead to a 3D aware VFM. Building on these findings, we develop several variants that achieve state-of-the-art across diverse datasets. This makes Feat2GS useful for probing VFMs, and as a simple-yet-effective baseline for novel-view synthesis. Code and data are available at https://fanegg.github.io/Feat2GS/.