# Generalized Shape Metrics on Neural Representations

arXiv: https://arxiv.org/abs/2110.14739 · HTML: https://mlsanity.com/paper/2110.14739/

## Key facts

- Cited by 112 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 68 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Topological Autoencoders](https://mlsanity.com/paper/1906.00722/) (2019)
- [On Linear Identifiability of Learned Representations](https://mlsanity.com/paper/2007.00810/) (2020)
- [Optimal whitening and decorrelation](https://mlsanity.com/paper/1512.00809/) (2015)
- [geomstats: a Python Package for Riemannian Geometry in Machine Learning](https://mlsanity.com/paper/1805.08308/) (2018)
- [Universality and individuality in neural dynamics across large populations of recurrent networks](https://mlsanity.com/paper/1907.08549/) (2019)
- [Geodesic Exponential Kernels: When Curvature and Linearity Conflict](https://mlsanity.com/paper/1411.0296/) (2014)
- [Neural tuning and representational geometry](https://mlsanity.com/paper/2104.09743/) (2021)
- [Minimum-Distortion Embedding](https://mlsanity.com/paper/2103.02559/) (2021)
- [Kernel method for persistence diagrams via kernel embedding and weight factor](https://mlsanity.com/paper/1706.03472/) (2017)
- [Analyzing biological and artificial neural networks: challenges with opportunities for synergy?](https://mlsanity.com/paper/1810.13373/) (2018)
- [Element-centric clustering comparison unifies overlaps and hierarchy](https://mlsanity.com/paper/1706.06136/) (2017)
- [A Tutorial on Canonical Correlation Methods](https://mlsanity.com/paper/1711.02391/) (2017)
- [Manifold GPLVMs for discovering non-Euclidean latent structure in neural data](https://mlsanity.com/paper/2006.07429/) (2020)
- [Levels of Analysis for Machine Learning](https://mlsanity.com/paper/2004.05107/) (2020)
- [Decoding of neural data using cohomological feature extraction](https://mlsanity.com/paper/1711.07205/) (2017)
- [A Mathematical Theory for Clustering in Metric Spaces](https://mlsanity.com/paper/1509.07755/) (2015)
- [A Framework for Shape Analysis via Hilbert Space Embedding](https://mlsanity.com/paper/1412.4174/) (2014)

## Built on by (most cited first, 30 of 112)

- [FlexiViT: One Model for All Patch Sizes](https://mlsanity.com/paper/2212.08013/) (2022)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [Deep Model Reassembly](https://mlsanity.com/paper/2210.17409/) (2022)
- [On the Symmetries of Deep Learning Models and their Internal Representations](https://mlsanity.com/paper/2205.14258/) (2022)
- [Latent Space Translation via Semantic Alignment](https://mlsanity.com/paper/2311.00664/) (2023)
- [Revisiting the Platonic Representation Hypothesis: An Aristotelian View](https://mlsanity.com/paper/2602.14486/) (2026)
- [The Tunnel Effect: Building Data Representations in Deep Neural Networks](https://mlsanity.com/paper/2305.19753/) (2023)
- [Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysis](https://mlsanity.com/paper/2306.10168/) (2023)
- [Deconfounded Representation Similarity for Comparison of Neural Networks](https://mlsanity.com/paper/2202.00095/) (2022)
- [ReSi: A Comprehensive Benchmark for Representational Similarity Measures](https://mlsanity.com/paper/2408.00531/) (2024)
- [Soft Matching Distance: A metric on neural representations that captures single-neuron tuning](https://mlsanity.com/paper/2311.09466/) (2023)
- [Objective drives the consistency of representational similarity across datasets](https://mlsanity.com/paper/2411.05561/) (2024)
- [What Representational Similarity Measures Imply about Decodable Information](https://mlsanity.com/paper/2411.08197/) (2024)
- [Representational dissimilarity metric spaces for stochastic neural networks](https://mlsanity.com/paper/2211.11665/) (2022)
- [Duality of Bures and Shape Distances with Implications for Comparing Neural Representations](https://mlsanity.com/paper/2311.11436/) (2023)
- [Brain-Model Evaluations Need the NeuroAI Turing Test](https://mlsanity.com/paper/2502.16238/) (2025)
- [Interpretable statistical representations of neural population dynamics and geometry](https://mlsanity.com/paper/2304.03376/) (2023)
- [Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory](https://mlsanity.com/paper/2310.06756/) (2023)
- [Understanding the Inner Workings of Language Models Through Representation Dissimilarity](https://mlsanity.com/paper/2310.14993/) (2023)
- [Evaluating Representational Similarity Measures from the Lens of Functional Correspondence](https://mlsanity.com/paper/2411.14633/) (2024)
- [Differentiable Optimization of Similarity Scores Between Models and Brains](https://mlsanity.com/paper/2407.07059/) (2024)
- [Neural Lineage](https://mlsanity.com/paper/2406.11129/) (2024)
- [The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions](https://mlsanity.com/paper/2506.13234/) (2025)
- [Discriminating image representations with principal distortions](https://mlsanity.com/paper/2410.15433/) (2024)
- [Graph-Based Similarity of Neural Network Representations](https://mlsanity.com/paper/2111.11165/) (2021)
- [Effective Layer Pruning Through Similarity Metric Perspective](https://mlsanity.com/paper/2405.17081/) (2024)
- [Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts](https://mlsanity.com/paper/2502.18710/) (2025)

## Abstract (arXiv)

Understanding the operation of biological and artificial networks remains a difficult and important challenge. To identify general principles, researchers are increasingly interested in surveying large collections of networks that are trained on, or biologically adapted to, similar tasks. A standardized set of analysis tools is now needed to identify how network-level covariates -- such as architecture, anatomical brain region, and model organism -- impact neural representations (hidden layer activations). Here, we provide a rigorous foundation for these analyses by defining a broad family of metric spaces that quantify representational dissimilarity. Using this framework we modify existing representational similarity measures based on canonical correlation analysis to satisfy the triangle inequality, formulate a novel metric that respects the inductive biases in convolutional layers, and identify approximate Euclidean embeddings that enable network representations to be incorporated into essentially any off-the-shelf machine learning method. We demonstrate these methods on large-scale datasets from biology (Allen Institute Brain Observatory) and deep learning (NAS-Bench-101). In doing so, we identify relationships between neural representations that are interpretable in terms of anatomical features and model performance.