# Differentiable Optimization of Similarity Scores Between Models and Brains

arXiv: https://arxiv.org/abs/2407.07059 · HTML: https://mlsanity.com/paper/2407.07059/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 24 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Builds on (most cited first, 6 of 6)

- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Generalized Shape Metrics on Neural Representations](https://mlsanity.com/paper/2110.14739/) (2021)
- [Task-Driven Convolutional Recurrent Models of the Visual System](https://mlsanity.com/paper/1807.00053/) (2018)
- [Similarity of Neural Networks with Gradients](https://mlsanity.com/paper/2003.11498/) (2020)
- [Duality of Bures and Shape Distances with Implications for Comparing Neural Representations](https://mlsanity.com/paper/2311.11436/) (2023)

## Built on by (most cited first, 10 of 10)

- [What Representational Similarity Measures Imply about Decodable Information](https://mlsanity.com/paper/2411.08197/) (2024)
- [Evaluating Representational Similarity Measures from the Lens of Functional Correspondence](https://mlsanity.com/paper/2411.14633/) (2024)
- [Model Alignment Search](https://mlsanity.com/paper/2501.06164/) (2025)
- [Estimating Neural Representation Alignment from Sparsely Sampled Inputs and Features](https://mlsanity.com/paper/2502.15104/) (2025)
- [Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families](https://mlsanity.com/paper/2509.04622/) (2025)
- [A Framework for Standardizing Similarity Measures in a Rapidly Evolving Field](https://mlsanity.com/paper/2409.18333/) (2024)
- [Can Biologically Plausible Temporal Credit Assignment Rules Match BPTT for Neural Similarity? E-prop as an Example](https://mlsanity.com/paper/2506.06904/) (2025)
- [Scalable and Interpretable Representation Alignment with Ordinal Similarity](https://mlsanity.com/paper/2606.16379/) (2026)
- [Comparing and Integrating Different Notions of Representational Correspondence in Neural Systems](https://mlsanity.com/paper/2509.21628/) (2025)
- [Knowledge distillation through geometry-aware representational alignment](https://mlsanity.com/paper/2509.25253/) (2025)

## Abstract (arXiv)

How do we know if two systems - biological or artificial - process information in a similar way? Similarity measures such as linear regression, Centered Kernel Alignment (CKA), Normalized Bures Similarity (NBS), and angular Procrustes distance, are often used to quantify this similarity. However, it is currently unclear what drives high similarity scores and even what constitutes a "good" score. Here, we introduce a novel tool to investigate these questions by differentiating through similarity measures to directly maximize the score. Surprisingly, we find that high similarity scores do not guarantee encoding task-relevant information in a manner consistent with neural data; and this is particularly acute for CKA and even some variations of cross-validated and regularized linear regression. We find no consistent threshold for a good similarity score - it depends on both the measure and the dataset. In addition, synthetic datasets optimized to maximize similarity scores initially learn the highest variance principal component of the target dataset, but some methods like angular Procrustes capture lower variance dimensions much earlier than methods like CKA. To shed light on this, we mathematically derive the sensitivity of CKA, angular Procrustes, and NBS to the variance of principal component dimensions, and explain the emphasis CKA places on high variance components. Finally, by jointly optimizing multiple similarity measures, we characterize their allowable ranges and reveal that some similarity measures are more constraining than others. While current measures offer a seemingly straightforward way to quantify the similarity between neural systems, our work underscores the need for careful interpretation. We hope the tools we developed will be used by practitioners to better understand current and future similarity measures.