# Similarity of Neural Networks with Gradients

arXiv: https://arxiv.org/abs/2003.11498 · HTML: https://mlsanity.com/paper/2003.11498/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2020.

## Built on by (most cited first, 16 of 16)

- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [Understanding the Dynamics of DNNs Using Graph Modularity](https://mlsanity.com/paper/2111.12485/) (2021)
- [ReSi: A Comprehensive Benchmark for Representational Similarity Measures](https://mlsanity.com/paper/2408.00531/) (2024)
- [Duality of Bures and Shape Distances with Implications for Comparing Neural Representations](https://mlsanity.com/paper/2311.11436/) (2023)
- [Latent Space Explanation by Intervention](https://mlsanity.com/paper/2112.04895/) (2021)
- [Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory](https://mlsanity.com/paper/2310.06756/) (2023)
- [Differentiable Optimization of Similarity Scores Between Models and Brains](https://mlsanity.com/paper/2407.07059/) (2024)
- [Graph-Based Similarity of Neural Network Representations](https://mlsanity.com/paper/2111.11165/) (2021)
- [Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent Spaces](https://mlsanity.com/paper/2503.05283/) (2025)
- [From Charts to Atlas: Merging Latent Spaces into One](https://mlsanity.com/paper/2311.06547/) (2023)
- [StitchNet: Composing Neural Networks from Pre-Trained Fragments](https://mlsanity.com/paper/2301.01947/) (2023)
- [FairBranch: Mitigating Bias Transfer in Fair Multi-task Learning](https://mlsanity.com/paper/2310.13746/) (2023)
- [A Framework for Standardizing Similarity Measures in a Rapidly Evolving Field](https://mlsanity.com/paper/2409.18333/) (2024)
- [Normalized Space Alignment: A Versatile Metric for Representation Analysis](https://mlsanity.com/paper/2411.04512/) (2024)

## Abstract (arXiv)

A suitable similarity index for comparing learnt neural networks plays an important role in understanding the behaviour of the highly-nonlinear functions, and can provide insights on further theoretical analysis and empirical studies. We define two key steps when comparing models: firstly, the representation abstracted from the learnt model, where we propose to leverage both feature vectors and gradient ones (which are largely ignored in prior work) into designing the representation of a neural network. Secondly, we define the employed similarity index which gives desired invariance properties, and we facilitate the chosen ones with sketching techniques for comparing various datasets efficiently. Empirically, we show that the proposed approach provides a state-of-the-art method for computing similarity of neural networks that are trained independently on different datasets and the tasks defined by the datasets.