The Cross-Architecture Substrate: A Domain-Transcendent, Calibration-Surviving Geometric Invariant of Modern Vision Encoders
Not on the map. Why?
Abstract (arXiv)
Different vision neural networks -- trained to classify, contrast, reconstruct, or match images to text -- should have correspondingly different internal representations. We report that they do not. After training, the top sixteen principal directions of variation inside thirteen modern vision encoders converge to the same sixteen-dimensional geometric object. We call this the cross-architecture substrate and study it with PCA, centred kernel alignment (CKA), and Pang 2026 calibration. The substrate transports across four visual domains (natural photographs, medical CT, satellite, microscopy) at median Procrustes-CKA 0.679, and across eight domains (adding sketches, depth, thermal infrared, astronomy) at 0.604, every pair >0.40. It survives Pang calibration globally (7.4x disc-vs-MAE separation, n=13,394) and locally (4.82-5.30, p<10^{-44}). It is not pixel statistics (0.263), not Gabor features (0.31), not a random projection (0.041), and emerges in the first 10% of training while accuracy keeps climbing. We deliver four applications: a label-free transferability filter beating LogME (3x faster, +0.15 Kendall-tau); a four-way domain detector (99.6% accuracy); a frozen low-shot probe (16 dims beat 768-dim DINOv2 by 3.78pp at N=50 labels per class); and a teacher-free distillation auxiliary matching trained-teacher KD on 33 pairs (7.56pp peak gain at 10% label fraction). The substrate does not cross modalities, does not help cross-paradigm distillation, and does not predict transfer quality (rho=0.08 against transfer accuracy).
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 5 Jun 2026.
Builds on
26 of 36 references.
References · 26
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2018DARTS: Differentiable Architecture Searchcited by 1,782
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2020Gradient Surgery for Multi-Task Learningcited by 1,210
- 2019Contrastive Representation Distillationcited by 734
- 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
- 2024The Platonic Representation Hypothesiscited by 362
- 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
- 2019Intrinsic dimension of data representations in deep neural networkscited by 276
- 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
- 2020Self-Knowledge Distillation with Progressive Refinement of Targetscited by 106
14 more
- 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
- 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
- 2024REEF: Representation Encoding Fingerprints for Large Language Modelscited by 34
- 2026Revisiting the Platonic Representation Hypothesis: An Aristotelian Viewcited by 33
- 2025The Universal Weight Subspace Hypothesiscited by 22
- 2024What Representational Similarity Measures Imply about Decodable Informationcited by 18
- 2025Activation Space Interventions Can Be Transferred Between Large Language Modelscited by 17
- 2024Universal dimensions of visual representationcited by 12
- 2024Unsupervised Model Tree Heritage Recoverycited by 11
- 2026Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scalecited by 10
- 2024Neural Lineagecited by 9
- 2026Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoningcited by 3
- 2025How NOT to benchmark your SITE metric: Beyond Static Leaderboards and Towards Realistic Evaluationcited by 2
- 2026IdEst: Assessing Self-Supervised Learning Representations via Intrinsic Dimensioncited by 2
Built on by
Not cited yet.