mlsanityData to 7 Oct 2026

Similarity of Neural Network Representations Revisited

arXiv 1905.00414 ↗First posted 1 May 2019cs.LG q-bio.NC stat.MLcited by 1,695

Not on the map. Why?

Cited by 53 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent work has sought to understand the behavior of neural networks by comparing representations between layers and between different trained models. We examine methods for comparing neural network representations based on canonical correlation analysis (CCA). We show that CCA belongs to a family of statistics for measuring multivariate similarity, but that neither CCA nor any other statistic that is invariant to invertible linear transformation can measure meaningful similarities between representations of higher dimension than the number of data points. We introduce a similarity index that measures the relationship between representational similarity matrices and does not suffer from this limitation. This similarity index is equivalent to centered kernel alignment (CKA) and is also closely connected to CCA. Unlike CCA, CKA can reliably identify correspondences between representations in networks trained from different initializations.

Key facts. Cited by 1,695 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 114; 2022: 159; 2023: 219; 2024: 255; 2025: 326; 2026: 622.

1,695 papers.

Sort:

By area:

  1. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  2. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 906
  3. 2021Robust fine-tuning of zero-shot modelscited by 785
  4. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  5. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  6. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  7. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  8. 2024The Platonic Representation Hypothesiscited by 362
  9. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  10. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  11. 2024TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variablescited by 241
  12. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
38 more
  1. 2021Does Knowledge Distillation Really Work?cited by 187
  2. 2022Patching open-vocabulary models by interpolating weightscited by 187
  3. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  4. 2023SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Modelcited by 178
  5. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  6. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  7. 2022Relative representations enable zero-shot latent space communicationcited by 175
  8. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 165
  9. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  10. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  11. 2022FlexiViT: One Model for All Patch Sizescited by 151
  12. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  13. 2023Getting aligned on representational alignmentcited by 142
  14. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  15. 2022Linearly Mapping from Image to Text Spacecited by 139
  16. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  17. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  18. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  19. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  20. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  21. 2024Model merging with SVD to tie the Knotscited by 118
  22. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  23. 2024TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysiscited by 113
  24. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 112
  25. 2021Generalized Shape Metrics on Neural Representationscited by 109
  26. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  27. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  28. 2024Universal Neurons in GPT2 Language Modelscited by 100
  29. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  30. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 95
  31. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  32. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  33. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  34. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
  35. 2023Modular Deep Learningcited by 82
  36. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  37. 2021A Broad Study on the Transferability of Visual Representations with Contrastive Learningcited by 78
  38. 2022When Do Flat Minima Optimizers Work?cited by 78

50 most cited of 1,695 All 1,695 (JSON)