Geometric Stability: The Missing Axis of Representations
Not on the map. Why?
Abstract (arXiv)
Representational similarity methods compare the geometries of neural representations, but they do not measure how consistently the geometry of a single representation is recovered from subsets of its feature coordinates. We call this property geometric stability and introduce Shesha, which estimates it by correlating representational dissimilarity matrices from complementary random feature subsets. Shesha is not invariant to orthogonal rotations: representations with identical Gram matrices, and therefore identical linear CKA, can have different geometric stability. Controlled transformations further separate the quantities. Across 2,463 encoder configurations spanning seven domains, similarity and stability are positively associated across non-PCA transformations (ρ=+0.75) but negatively associated under PCA-coordinate compression (ρ=-0.47). We further evaluate 170 pretrained vision models across six datasets. DINOv2 combines strong transfer performance with bottom-quartile stability on five of six datasets, showing that transferability and feature-split stability need not coincide. Across random feature subsets, the marginal relationship between Shesha and linear-probe variability is dataset-dependent; after controlling for task alignment with LogME, higher Shesha is associated with lower variability on five of six datasets. These results identify geometric stability as a basis-dependent property that complements representational similarity and task alignment.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 75 references that are in the map. First posted on arXiv 14 Jan 2026.
Builds on
22 of 75 references.
References · 22
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2013Describing Textures in the Wildcited by 2,110
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
- 2021Robust fine-tuning of zero-shot modelscited by 785
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
- 2018Do Better ImageNet Models Transfer Better?cited by 560
- 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
- 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
- 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
10 more
- 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 182
- 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
- 2023Getting aligned on representational alignmentcited by 142
- 2019Veridical Data Sciencecited by 65
- 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 33
- 2021Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubscited by 28
- 2024Dimensions underlying the representational alignment of deep neural networks with humanscited by 22
- 2018Procrustes Metrics on Covariance Operators and Optimal Transportation of Gaussian Processescited by 19
- 2023The Topology and Geometry of Neural Representationscited by 13
- 2024Bayesian Comparisons Between Representationscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.