# Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation

arXiv: https://arxiv.org/abs/1810.11750 · HTML: https://mlsanity.com/paper/1810.11750/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2018.

## Built on by (most cited first, 30 of 62)

- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [Similarity and Matching of Neural Network Representations](https://mlsanity.com/paper/2110.14633/) (2021)
- [On the geometry of generalization and memorization in deep neural networks](https://mlsanity.com/paper/2105.14602/) (2021)
- [Representation Topology Divergence: A Method for Comparing Neural Network Representations](https://mlsanity.com/paper/2201.00058/) (2021)
- [On the Symmetries of Deep Learning Models and their Internal Representations](https://mlsanity.com/paper/2205.14258/) (2022)
- [Harnessing the Universal Geometry of Embeddings](https://mlsanity.com/paper/2505.12540/) (2025)
- [Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)](https://mlsanity.com/paper/2110.05922/) (2021)
- [Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization](https://mlsanity.com/paper/2305.02099/) (2023)
- [Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures](https://mlsanity.com/paper/2410.06672/) (2024)
- [Grounding Representation Similarity with Statistical Testing](https://mlsanity.com/paper/2108.01661/) (2021)
- [Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2410.06981/) (2024)
- [Understanding the Dynamics of DNNs Using Graph Modularity](https://mlsanity.com/paper/2111.12485/) (2021)
- [SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models](https://mlsanity.com/paper/2410.14720/) (2024)
- [Random initialisations performing above chance and how to find them](https://mlsanity.com/paper/2209.07509/) (2022)
- [GULP: a prediction-based metric between representations](https://mlsanity.com/paper/2210.06545/) (2022)
- [CoReS: Compatible Representations via Stationarity](https://mlsanity.com/paper/2111.07632/) (2021)
- [To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance](https://mlsanity.com/paper/2511.12121/) (2025)
- [Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory](https://mlsanity.com/paper/2310.06756/) (2023)
- [When Deep Classifiers Agree: Analyzing Correlations between Learning Order and Image Statistics](https://mlsanity.com/paper/2105.08997/) (2021)
- [Measuring Representational Robustness of Neural Networks Through Shared Invariances](https://mlsanity.com/paper/2206.11939/) (2022)
- [Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradient](https://mlsanity.com/paper/2304.13098/) (2023)
- [Representation Potentials of Foundation Models for Multimodal Alignment: A Survey](https://mlsanity.com/paper/2510.05184/) (2025)
- [How do Variational Autoencoders Learn? Insights from Representational Similarity](https://mlsanity.com/paper/2205.08399/) (2022)
- [Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation](https://mlsanity.com/paper/2308.09105/) (2023)
- [Formation of Representations in Neural Networks](https://mlsanity.com/paper/2410.03006/) (2024)

## Abstract (arXiv)

It is widely believed that learning good representations is one of the main reasons for the success of deep neural networks. Although highly intuitive, there is a lack of theory and systematic approach quantitatively characterizing what representations do deep neural networks learn. In this work, we move a tiny step towards a theory and better understanding of the representations. Specifically, we study a simpler problem: How similar are the representations learned by two networks with identical architecture but trained from different initializations. We develop a rigorous theory based on the neuron activation subspace match model. The theory gives a complete characterization of the structure of neuron activation subspace matches, where the core concepts are maximum match and simple match which describe the overall and the finest similarity between sets of neurons in two networks respectively. We also propose efficient algorithms to find the maximum match and simple matches. Finally, we conduct extensive experiments using our algorithms. Experimental results suggest that, surprisingly, representations learned by the same convolutional layers of networks trained from different initializations are not as similar as prevalently expected, at least in terms of subspace match.