# Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention

arXiv: https://arxiv.org/abs/2607.01987 · HTML: https://mlsanity.com/paper/2607.01987/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 47 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 15 of 15)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [Deep ViT Features as Dense Visual Descriptors](https://mlsanity.com/paper/2112.05814/) (2021)
- [The Intrinsic Dimension of Images and Its Impact on Learning](https://mlsanity.com/paper/2104.08894/) (2021)
- [What Do Self-Supervised Vision Transformers Learn?](https://mlsanity.com/paper/2305.00729/) (2023)
- [Into the Rabbit Hull: From Task-Relevant Concepts in DINO to Minkowski Geometry](https://mlsanity.com/paper/2510.08638/) (2025)
- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [Learning on the Manifold: Unlocking Standard Diffusion Transformers with Representation Encoders](https://mlsanity.com/paper/2602.10099/) (2026)
- [A Survey of the Self Supervised Learning Mechanisms for Vision Transformers](https://mlsanity.com/paper/2408.17059/) (2024)
- [The Spatial Blindspot of Vision-Language Models](https://mlsanity.com/paper/2601.09954/) (2026)
- [Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation](https://mlsanity.com/paper/2403.05056/) (2024)
- [Learning to Compress: Local Rank and Information Compression in Deep Neural Networks](https://mlsanity.com/paper/2410.07687/) (2024)
- [Mechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AI](https://mlsanity.com/paper/2503.18762/) (2025)

## Built on by (most cited first, 2 of 2)

- [Emergent Multi-View Geometry Through Self-Distillation](https://mlsanity.com/paper/2609.39227/) (2026)
- [LoDEOT: Low-Dimensional and Efficient Offset Tokens for Building Footprint Extraction from Off-Nadir Imagery](https://mlsanity.com/paper/2610.05899/) (2026)

## Abstract (arXiv)

We introduce a controlled subspace intervention framework to investigate how self-supervised Vision Transformers (ViTs) encode dense geometric information. While linear probing is widely used to assess geometric representations, it treats features as a black box, failing to disentangle the underlying topology. To address this issue, we decompose the weights of converged linear probes to isolate the low-rank subspaces containing explicit geometric signals using Singular Value Decomposition (SVD). Our perspective yields three key insights: (1) Pre-training objectives determine how features are encoded. DINOv2 aligns spatial features for efficient linear extraction, while Masked Autoencoders (MAE) tend to disperse these signals, requiring a broader spatial context. (2) Explicit geometric representations are highly compressible, suggesting dense predictive heads could potentially be constrained to low-rank subspaces with minimal performance loss. (3) The layer-wise task affinity suggests that geometric precision peaks at intermediate layers before yielding to semantic abstraction in the final layers. By connecting internal encoding mechanics with downstream performance, these findings provide a basis for effective feature selection and lightweight decoder design. The source code is available at https://github.com/Zhou-Weichen/Geosubprobe.