# How not to Stitch Representations to Measure Similarity: Task Loss Matching versus Direct Matching

arXiv: https://arxiv.org/abs/2412.11299 · HTML: https://mlsanity.com/paper/2412.11299/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [Revisiting Model Stitching to Compare Neural Representations](https://mlsanity.com/paper/2106.07682/) (2021)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Offline bilingual word vectors, orthogonal transformations and the inverted softmax](https://mlsanity.com/paper/1702.03859/) (2017)
- [Similarity and Matching of Neural Network Representations](https://mlsanity.com/paper/2110.14633/) (2021)
- [Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners](https://mlsanity.com/paper/2306.00561/) (2023)
- [Model Stitching: Looking For Functional Similarity Between Representations](https://mlsanity.com/paper/2303.11277/) (2023)
- [Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations](https://mlsanity.com/paper/2306.17105/) (2023)
- [Stitched ViTs are Flexible Vision Backbones](https://mlsanity.com/paper/2307.00154/) (2023)
- [StitchNet: Composing Neural Networks from Pre-Trained Fragments](https://mlsanity.com/paper/2301.01947/) (2023)
- [Stitching for Neuroevolution: Recombining Deep Neural Networks without Breaking Them](https://mlsanity.com/paper/2403.14224/) (2024)

## Built on by (most cited first, 5 of 5)

- [Revisiting Model Stitching In the Foundation Model Era](https://mlsanity.com/paper/2603.12433/) (2026)
- [KLAS: Using Similarity to Stitch Neural Networks for Improved Accuracy-Efficiency Tradeoffs](https://mlsanity.com/paper/2605.29259/) (2026)
- [Sharing Knowledge without Sharing Data: Stitches can improve ensembles of disjointly trained models](https://mlsanity.com/paper/2512.17592/) (2025)
- [Deep Minds and Shallow Probes](https://mlsanity.com/paper/2605.11448/) (2026)
- [A Stitch in Time Saves Nine: Preserving Policy Compatibility Under Perception Updates in End-to-End Autonomous Driving](https://mlsanity.com/paper/2606.21509/) (2026)

## Abstract (arXiv)

Measuring the similarity of the internal representations of deep neural networks is an important and challenging problem. Model stitching has been proposed as a possible approach, where two half-networks are connected by mapping the output of the first half-network to the input of the second one. The representations are considered functionally similar if the resulting stitched network achieves good task-specific performance. The mapping is normally created by training an affine stitching layer on the task at hand while freezing the two half-networks, a method called task loss matching. Here, we argue that task loss matching may be very misleading as a similarity index. For example, it can indicate very high similarity between very distant layers, whose representations are known to have different functional properties. Moreover, it can indicate very distant layers to be more similar than architecturally corresponding layers. Even more surprisingly, when comparing layers within the same network, task loss matching often indicates that some layers are more similar to a layer than itself. We argue that the main reason behind these problems is that task loss matching tends to create out-of-distribution representations to improve task-specific performance. We demonstrate that direct matching (when the mapping minimizes the distance between the stitched representations) does not suffer from these problems. We compare task loss matching, direct matching, and well-known similarity indices such as CCA and CKA. We conclude that direct matching strikes a good balance between the structural and functional requirements for a good similarity index.