# Similarity and Matching of Neural Network Representations

arXiv: https://arxiv.org/abs/2110.14633 · HTML: https://mlsanity.com/paper/2110.14633/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [Linear Mode Connectivity in Multitask and Continual Learning](https://mlsanity.com/paper/2010.04495/) (2020)
- [Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation](https://mlsanity.com/paper/1810.11750/) (2018)
- [Similarity Analysis of Contextual Word Representation Models](https://mlsanity.com/paper/2005.01172/) (2020)

## Built on by (most cited first, 30 of 78)

- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Analyzing Transformers in Embedding Space](https://mlsanity.com/paper/2209.02535/) (2022)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Deep Model Reassembly](https://mlsanity.com/paper/2210.17409/) (2022)
- [Identifiability of deep generative models without auxiliary information](https://mlsanity.com/paper/2206.10044/) (2022)
- [On the Symmetries of Deep Learning Models and their Internal Representations](https://mlsanity.com/paper/2205.14258/) (2022)
- [Text-To-Concept (and Back) via Cross-Model Alignment](https://mlsanity.com/paper/2305.06386/) (2023)
- [Demystifying Verbatim Memorization in Large Language Models](https://mlsanity.com/paper/2407.17817/) (2024)
- [Latent Space Translation via Semantic Alignment](https://mlsanity.com/paper/2311.00664/) (2023)
- [T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching](https://mlsanity.com/paper/2402.14167/) (2024)
- [Training-Free Pretrained Model Merging](https://mlsanity.com/paper/2403.01753/) (2024)
- [Understanding the Emergence of Multimodal Representation Alignment](https://mlsanity.com/paper/2502.16282/) (2025)
- [ModelDiff: A Framework for Comparing Learning Algorithms](https://mlsanity.com/paper/2211.12491/) (2022)
- [Stitchable Neural Networks](https://mlsanity.com/paper/2302.06586/) (2023)
- [Configurable Foundation Models: Building LLMs from a Modular Perspective](https://mlsanity.com/paper/2409.02877/) (2024)
- [Objective drives the consistency of representational similarity across datasets](https://mlsanity.com/paper/2411.05561/) (2024)
- [Deep Incubation: Training Large Models by Divide-and-Conquering](https://mlsanity.com/paper/2212.04129/) (2022)
- [Understanding Robust Learning through the Lens of Representation Similarities](https://mlsanity.com/paper/2206.09868/) (2022)
- [Representation Potentials of Foundation Models for Multimodal Alignment: A Survey](https://mlsanity.com/paper/2510.05184/) (2025)
- [Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator](https://mlsanity.com/paper/2510.13454/) (2025)
- [Latent Space Translation via Inverse Relative Projection](https://mlsanity.com/paper/2406.15057/) (2024)
- [Data Duplication: A Novel Multi-Purpose Attack Paradigm in Machine Unlearning](https://mlsanity.com/paper/2501.16663/) (2025)
- [Bootstrapping Parallel Anchors for Relative Representations](https://mlsanity.com/paper/2303.00721/) (2023)
- [Stitched ViTs are Flexible Vision Backbones](https://mlsanity.com/paper/2307.00154/) (2023)
- [Graph-Based Similarity of Neural Network Representations](https://mlsanity.com/paper/2111.11165/) (2021)
- [Measuring the Instability of Fine-Tuning](https://mlsanity.com/paper/2302.07778/) (2023)
- [Building Variable-sized Models via Learngene Pool](https://mlsanity.com/paper/2312.05743/) (2023)

## Abstract (arXiv)

We employ a toolset -- dubbed Dr. Frankenstein -- to analyse the similarity of representations in deep neural networks. With this toolset, we aim to match the activations on given layers of two trained neural networks by joining them with a stitching layer. We demonstrate that the inner representations emerging in deep convolutional neural networks with the same architecture but different initializations can be matched with a surprisingly high degree of accuracy even with a single, affine stitching layer. We choose the stitching layer from several possible classes of linear transformations and investigate their performance and properties. The task of matching representations is closely related to notions of similarity. Using this toolset, we also provide a novel viewpoint on the current line of research regarding similarity indices of neural network representations: the perspective of the performance on a task.