# GULP: a prediction-based metric between representations

arXiv: https://arxiv.org/abs/2210.06545 · HTML: https://mlsanity.com/paper/2210.06545/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 53 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [Feature Learning in Infinite-Width Neural Networks](https://mlsanity.com/paper/2011.14522/) (2020)
- [Neural Architecture Optimization](https://mlsanity.com/paper/1808.07233/) (2018)
- [Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?](https://mlsanity.com/paper/2005.00719/) (2020)
- [Offline bilingual word vectors, orthogonal transformations and the inverted softmax](https://mlsanity.com/paper/1702.03859/) (2017)
- [BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance](https://mlsanity.com/paper/1911.02969/) (2019)
- [Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation](https://mlsanity.com/paper/1810.11750/) (2018)
- [Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level](https://mlsanity.com/paper/2105.06020/) (2021)
- [Deconfounded Representation Similarity for Comparison of Neural Networks](https://mlsanity.com/paper/2202.00095/) (2022)
- [Knowledge Consistency between Neural Networks and Beyond](https://mlsanity.com/paper/1908.01581/) (2019)

## Built on by (most cited first, 17 of 17)

- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [Model Spider: Learning to Rank Pre-Trained Models Efficiently](https://mlsanity.com/paper/2306.03900/) (2023)
- [Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures](https://mlsanity.com/paper/2410.06672/) (2024)
- [ReSi: A Comprehensive Benchmark for Representational Similarity Measures](https://mlsanity.com/paper/2408.00531/) (2024)
- [What Representational Similarity Measures Imply about Decodable Information](https://mlsanity.com/paper/2411.08197/) (2024)
- [Towards a theory of model distillation](https://mlsanity.com/paper/2403.09053/) (2024)
- [On Affine Homotopy between Language Encoders](https://mlsanity.com/paper/2406.02329/) (2024)
- [Geometry-aware similarity metrics for neural representations on Riemannian and statistical manifolds](https://mlsanity.com/paper/2603.28764/) (2026)
- [Diagnosing Neural Convergence with Topological Alignment Spectra](https://mlsanity.com/paper/2411.08687/) (2024)
- [Manifold Approximation leads to Robust Kernel Alignment](https://mlsanity.com/paper/2510.22953/) (2025)
- [GIST: Generated Inputs Sets Transferability in Deep Learning](https://mlsanity.com/paper/2311.00801/) (2023)
- [Bayesian Comparisons Between Representations](https://mlsanity.com/paper/2411.08739/) (2024)
- [Explaining How Visual, Textual and Multimodal Encoders Share Concepts](https://mlsanity.com/paper/2507.18512/) (2025)
- [Normalized Space Alignment: A Versatile Metric for Representation Analysis](https://mlsanity.com/paper/2411.04512/) (2024)
- [Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?](https://mlsanity.com/paper/2607.17425/) (2026)
- [Hub-Spectral Activation of Latent Multimodal Knowledge](https://mlsanity.com/paper/2609.17094/) (2026)

## Abstract (arXiv)

Comparing the representations learned by different neural networks has recently emerged as a key tool to understand various architectures and ultimately optimize them. In this work, we introduce GULP, a family of distance measures between representations that is explicitly motivated by downstream predictive tasks. By construction, GULP provides uniform control over the difference in prediction performance between two representations, with respect to regularized linear prediction tasks. Moreover, it satisfies several desirable structural properties, such as the triangle inequality and invariance under orthogonal transformations, and thus lends itself to data embedding and visualization. We extensively evaluate GULP relative to other methods, and demonstrate that it correctly differentiates between architecture families, converges over the course of training, and captures generalization performance on downstream linear tasks.