# ReSi: A Comprehensive Benchmark for Representational Similarity Measures

arXiv: https://arxiv.org/abs/2408.00531 · HTML: https://mlsanity.com/paper/2408.00531/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 64 references that are in the map.
- First posted on arXiv 1 Aug 2024.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Revisiting Semi-Supervised Learning with Graph Embeddings](https://mlsanity.com/paper/1603.08861/) (2016)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [DropEdge: Towards Deep Graph Convolutional Networks on Node Classification](https://mlsanity.com/paper/1907.10903/) (2019)
- [GraphSAINT: Graph Sampling Based Inductive Learning Method](https://mlsanity.com/paper/1907.04931/) (2019)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change](https://mlsanity.com/paper/1605.09096/) (2016)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Position-aware Graph Neural Networks](https://mlsanity.com/paper/1906.04817/) (2019)
- [Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics](https://mlsanity.com/paper/2007.07400/) (2020)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)

## Built on by (most cited first, 19 of 19)

- [Revisiting the Platonic Representation Hypothesis: An Aristotelian View](https://mlsanity.com/paper/2602.14486/) (2026)
- [Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2410.06981/) (2024)
- [What Representational Similarity Measures Imply about Decodable Information](https://mlsanity.com/paper/2411.08197/) (2024)
- [Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers](https://mlsanity.com/paper/2412.06639/) (2024)
- [Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models](https://mlsanity.com/paper/2506.00653/) (2025)
- [Statistical and structural identifiability in representation learning](https://mlsanity.com/paper/2603.11970/) (2026)
- [Layer-wise LoRA fine-tuning: a similarity metric approach](https://mlsanity.com/paper/2602.05988/) (2026)
- [Manifold Approximation leads to Robust Kernel Alignment](https://mlsanity.com/paper/2510.22953/) (2025)
- [From Associations to Activations: Comparing Behavioral and Hidden-State Semantic Geometry in LLMs](https://mlsanity.com/paper/2602.00628/) (2026)
- [Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces](https://mlsanity.com/paper/2602.19367/) (2026)
- [Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments](https://mlsanity.com/paper/2504.07965/) (2025)
- [Revisiting the Relation Between Robustness and Universality](https://mlsanity.com/paper/2510.19427/) (2025)
- [The Impact of Dimensionality on the Stability of Node Embeddings](https://mlsanity.com/paper/2604.08492/) (2026)
- [From Layers to Networks: Comparing Neural Representations via Diffusion Geometry](https://mlsanity.com/paper/2605.15901/) (2026)
- [TopoAlign: Topology-Aware Visual Representation Alignment](https://mlsanity.com/paper/2605.25541/) (2026)
- [Representation Measurements Under Function-Preserving Reparameterizations](https://mlsanity.com/paper/2608.27020/) (2026)
- [When Pair Count Is Not the Sample Size: What All-Pairs Agent Comparisons Estimate](https://mlsanity.com/paper/2609.33012/) (2026)
- [Hidden Activations are not Enough I: Knowledge Matrices as Higher Representations](https://mlsanity.com/paper/2609.34166/) (2026)
- [Predicting Multi-View Rashomon Representation: Can We Learn Where Models Disagree?](https://mlsanity.com/paper/2609.39848/) (2026)

## Abstract (arXiv)

Measuring the similarity of different representations of neural architectures is a fundamental task and an open research challenge for the machine learning community. This paper presents the first comprehensive benchmark for evaluating representational similarity measures based on well-defined groundings of similarity. The representational similarity (ReSi) benchmark consists of (i) six carefully designed tests for similarity measures, (ii) 24 similarity measures, (iii) 14 neural network architectures, and (iv) seven datasets, spanning over the graph, language, and vision domains. The benchmark opens up several important avenues of research on representational similarity that enable novel explorations and applications of neural architectures. We demonstrate the utility of the ReSi benchmark by conducting experiments on various neural network architectures, real world datasets and similarity measures. All components of the benchmark are publicly available and thereby facilitate systematic reproduction and production of research results. The benchmark is extensible, future research can build on and further expand it. We believe that the ReSi benchmark can serve as a sound platform catalyzing future research that aims to systematically evaluate existing and explore novel ways of comparing representations of neural architectures.