mlsanityData to 7 Oct 2026

Revisiting Model Stitching to Compare Neural Representations

arXiv 2106.07682 ↗First posted 14 Jun 2021cs.LG stat.MLcited by 203

Not on the map. Why?

Abstract (arXiv)

We revisit and extend model stitching (Lenc & Vedaldi 2015) as a methodology to study the internal representations of neural networks. Given two trained and frozen models A and B, we consider a "stitched model'' formed by connecting the bottom-layers of A to the top-layers of B, with a simple trainable layer between them. We argue that model stitching is a powerful and perhaps under-appreciated tool, which reveals aspects of representations that measures such as centered kernel alignment (CKA) cannot. Through extensive experiments, we use model stitching to obtain quantitative verifications for intuitive statements such as "good networks learn similar representations'', by demonstrating that good networks of the same architecture, but trained in very different ways (e.g.: supervised vs. self-supervised learning), can be stitched to each other without drop in performance. We also give evidence for the intuition that "more is better'' by showing that representations learnt with (1) more data, (2) bigger width, or (3) more training time can be "plugged in'' to weaker models to improve performance. Finally, our experiments reveal a new structural property of SGD which we call "stitching connectivity'', akin to mode-connectivity: typical minima reached by SGD can all be stitched to each other with minimal change in accuracy.

Key facts. Cited by 203 papers in the map (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

22 of 35 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  8. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  9. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  10. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  11. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  12. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
10 more
  1. 2018Insights on representational similarity in neural networks with canonical correlationcited by 318
  2. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  3. 2014Understanding image representations by measuring their equivariance and equivalencecited by 240
  4. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  5. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  6. 2018Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representationcited by 62
  7. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  8. 2020Distributional Generalization: A New Kind of Generalizationcited by 41
  9. 2020On the surprising similarities between supervised and self-supervised modelscited by 29
  10. 2020For self-supervised learning, Rationality implies generalization, provablycited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 19; 2023: 28; 2024: 31; 2025: 42; 2026: 80.

203 papers.

Sort:

By area:

  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2024The Platonic Representation Hypothesiscited by 362
  4. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  5. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  6. 2022Relative representations enable zero-shot latent space communicationcited by 175
  7. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  8. 2022Linearly Mapping from Image to Text Spacecited by 139
  9. 2022Analyzing Transformers in Embedding Spacecited by 131
  10. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  11. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  12. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
38 more
  1. 2024Universal Neurons in GPT2 Language Modelscited by 100
  2. 2022Deep Model Reassemblycited by 78
  3. 2021Similarity and Matching of Neural Network Representationscited by 77
  4. 2022Identifiability of deep generative models without auxiliary informationcited by 70
  5. 2022On the Symmetries of Deep Learning Models and their Internal Representationscited by 63
  6. 2023Text-To-Concept (and Back) via Cross-Model Alignmentcited by 63
  7. 2025Harnessing the Universal Geometry of Embeddingscited by 54
  8. 2023Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivitycited by 50
  9. 2023Latent Space Translation via Semantic Alignmentcited by 46
  10. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  11. 2025We're Different, We're the Same: Creative Homogeneity Across LLMscited by 41
  12. 2025Shared Global and Local Geometry of Language Model Embeddingscited by 34
  13. 2024T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitchingcited by 32
  14. 2024Training-Free Pretrained Model Mergingcited by 32
  15. 2023Towards Personalized Federated Learning via Heterogeneous Model Reassemblycited by 31
  16. 2022ModelDiff: A Framework for Comparing Learning Algorithmscited by 30
  17. 2025Understanding the Emergence of Multimodal Representation Alignmentcited by 28
  18. 2023Stitchable Neural Networkscited by 26
  19. 2024On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigmcited by 25
  20. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 25
  21. 2024A deeper look at depth pruning of LLMscited by 23
  22. 2023Layer-wise Linear Mode Connectivitycited by 22
  23. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  24. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  25. 2023Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Modelscited by 20
  26. 2024Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIPcited by 19
  27. 2021Pretrained Language Models are Symbolic Mathematics Solvers too!cited by 17
  28. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  29. 2025Transferring Linear Features Across Language Models With Model Stitchingcited by 17
  30. 2024Objective drives the consistency of representational similarity across datasetscited by 16
  31. 2023Exploring the Relationship between In-Context Learning and Instruction Tuningcited by 14
  32. 2022Deep Incubation: Training Large Models by Divide-and-Conqueringcited by 13
  33. 2022Understanding Robust Learning through the Lens of Representation Similaritiescited by 12
  34. 2023Understanding the Inner Workings of Language Models Through Representation Dissimilaritycited by 12
  35. 2024Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval Augmented Generation Systemscited by 12
  36. 2025Aligning Multimodal Representations through an Information Bottleneckcited by 12
  37. 2022Measuring Representational Robustness of Neural Networks Through Shared Invariancescited by 11
  38. 2024Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignmentcited by 11

50 most cited of 203 All 203 (JSON)