mlsanityData to 14 Oct 2026

Similarity of Neural Network Models: A Survey of Functional and Representational Measures

arXiv 2305.06329 ↗First posted 10 May 2023cs.LGcited by 131

Not on the map. Why?

Abstract (arXiv)

Measuring similarity of neural networks to understand and improve their behavior has become an issue of great importance and research interest. In this survey, we provide a comprehensive overview of two complementary perspectives of measuring neural network similarity: (i) representational similarity, which considers how activations of intermediate layers differ, and (ii) functional similarity, which considers how models differ in their outputs. In addition to providing detailed descriptions of existing measures, we summarize and discuss results on the properties of and relationships between these measures, and point to open research problems. We hope our work lays a foundation for more systematic research on the properties and applicability of similarity measures for neural network models.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Builds on 128 of its 261 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

128 of 261 references.

References · 128

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2017Graph Attention Networkscited by 8,923
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  10. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  11. 2017Inductive Representation Learning on Large Graphscited by 6,320
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
38 more
  1. 2014Going Deeper with Convolutionscited by 4,241
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  4. 2022A ConvNet for the 2020scited by 3,579
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  9. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  10. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  12. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  13. 2020Knowledge Distillation: A Surveycited by 1,603
  14. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  15. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  16. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  17. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  18. 2020Designing Network Design Spacescited by 960
  19. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  20. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  21. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  22. 2019Relational Knowledge Distillationcited by 889
  23. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  24. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  25. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  26. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  27. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  28. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  29. 2013Do Deep Nets Really Need to be Deep?cited by 474
  30. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  31. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
  32. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  33. 2020Understanding the Behaviour of Contrastive Losscited by 398
  34. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  35. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
  36. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  37. 2020Evaluation of Text Generation: A Surveycited by 277
  38. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 245

50 most cited of 128 All 128 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 32; 2025: 38; 2026: 48.

131 papers.

Sort:

By area:

  1. 2024The Platonic Representation Hypothesiscited by 382
  2. 2023Getting aligned on representational alignmentcited by 143
  3. 2024Universal Neurons in GPT2 Language Modelscited by 102
  4. 2026Revisiting the Platonic Representation Hypothesis: An Aristotelian Viewcited by 44
  5. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  6. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architecturescited by 31
  7. 2025Understanding the Emergence of Multimodal Representation Alignmentcited by 31
  8. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  9. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 27
  10. 2023From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communicationcited by 21
  11. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 21
  12. 2024ReSi: A Comprehensive Benchmark for Representational Similarity Measurescited by 19
38 more
  1. 2024Objective drives the consistency of representational similarity across datasetscited by 18
  2. 2024What Representational Similarity Measures Imply about Decodable Informationcited by 18
  3. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 17
  4. 2023Duality of Bures and Shape Distances with Implications for Comparing Neural Representationscited by 15
  5. 2025Aligning Multimodal Representations through an Information Bottleneckcited by 13
  6. 2023Understanding the Inner Workings of Language Models Through Representation Dissimilaritycited by 12
  7. 2024Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systemscited by 12
  8. 2024Persistent Topological Features in Large Language Modelscited by 12
  9. 2024Evaluating Representational Similarity Measures from the Lens of Functional Correspondencecited by 12
  10. 2025Layers at Similar Depths Generate Similar Activations Across LLM Architecturescited by 12
  11. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 11
  12. 2024Discriminating image representations with principal distortionscited by 8
  13. 2025Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Modelscited by 7
  14. 2025The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Modelscited by 7
  15. 2026Differential syntactic and semantic encoding in LLMscited by 7
  16. 2023Towards Measuring Representational Similarity of Large Language Modelscited by 6
  17. 2025Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent Spacescited by 6
  18. 2023CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Modelcited by 5
  19. 2024Mapping the Multiverse of Latent Representationscited by 5
  20. 2025Representational Similarity via Interpretable Visual Conceptscited by 5
  21. 2025Position: We Need An Algorithmic Understanding of Generative AIcited by 5
  22. 2025Toward a Theory of Generalizability in LLM Mechanistic Interpretability Researchcited by 5
  23. 2026Multi-Way Representation Alignmentcited by 5
  24. 2023Estimating Shape Distances on Neural Representations with Limited Samplescited by 4
  25. 2024Intrinsic Dimension Correlation: uncovering nonlinear connections in multimodal representationscited by 4
  26. 2024Training the Untrainable: Introducing Inductive Bias via Representational Alignmentcited by 4
  27. 2024Comparing noisy neural population dynamics using optimal transport distancescited by 4
  28. 2025Connecting Neural Models Latent Geometries with Relative Geodesic Representationscited by 4
  29. 2025When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspectivecited by 4
  30. 2025Optimal Transport Depth Up-Scalingcited by 4
  31. 2026Ventriloquist LLMs: Linear Alignment of Late-Stage Representationscited by 4
  32. 2023From Charts to Atlas: Merging Latent Spaces into Onecited by 3
  33. 2024On the generalization capacity of neural networks during generic multimodal reasoningcited by 3
  34. 2024On Affine Homotopy between Language Encoderscited by 3
  35. 2024TOAST: Transformer Optimization using Adaptive and Simple Transformationscited by 3
  36. 2025Model Alignment Searchcited by 3
  37. 2025Causes and Consequences of Representational Similarity in Machine Learning Modelscited by 3
  38. 2025Contextual Representation Flow Across Depth in State-Space and Transformer Architecturescited by 3

50 most cited of 131 All 131 (JSON)