# Comparing Foundation Models using Data Kernels

arXiv: https://arxiv.org/abs/2305.05126 · HTML: https://mlsanity.com/paper/2305.05126/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Statistical inference on random dot product graphs: a survey](https://mlsanity.com/paper/1709.05454/) (2017)
- [Mental State Classification Using Multi-graph Features](https://mlsanity.com/paper/2203.00516/) (2022)
- [Metric recovery from directed unweighted graphs](https://mlsanity.com/paper/1411.5720/) (2014)
- [Dynamic Silos: Increased Modularity in Intra-organizational Communication Networks during the Covid-19 Pandemic](https://mlsanity.com/paper/2104.00641/) (2021)
- [DBpedia NIF: Open, Large-Scale and Multilingual Knowledge Extraction Corpus](https://mlsanity.com/paper/1812.10315/) (2018)

## Built on by (most cited first, 14 of 14)

- [Tracking the perspectives of interacting language models](https://mlsanity.com/paper/2406.11938/) (2024)
- [Consistent estimation of generative model representations in the data kernel perspective space](https://mlsanity.com/paper/2409.17308/) (2024)
- [Statistical inference on black-box generative models in the data kernel perspective space](https://mlsanity.com/paper/2410.01106/) (2024)
- [A Statistical Turing Test for Generative Models](https://mlsanity.com/paper/2309.08913/) (2023)
- [Investigating social alignment via mirroring in a system of interacting language models](https://mlsanity.com/paper/2412.06834/) (2024)
- [Weight Tying Biases Token Embeddings Towards the Output Space](https://mlsanity.com/paper/2603.26663/) (2026)
- [Development and Application of Self-Supervised Machine Learning for Smoke Plume and Active Fire Identification from the FIREX-AQ Datasets](https://mlsanity.com/paper/2501.15343/) (2025)
- [Query-efficient model evaluation using cached responses](https://mlsanity.com/paper/2605.07096/) (2026)
- [Concentration bounds on response-based vector embeddings of black-box generative models](https://mlsanity.com/paper/2511.08307/) (2025)
- [Black-box model classification under the discriminative factorization](https://mlsanity.com/paper/2605.07878/) (2026)
- [Control Charts for Multi-agent Systems](https://mlsanity.com/paper/2605.11135/) (2026)
- [Fusing Multi- and Hyperspectral Satellite Data for Harmful Algal Bloom Monitoring with Self-Supervised and Hierarchical Deep Learning](https://mlsanity.com/paper/2510.02763/) (2025)
- [The View From Space: Navigating Instrumentation Differences with EOFMs](https://mlsanity.com/paper/2510.03316/) (2025)
- [Jailbreak susceptibility prediction and mitigation via the behavioral geometry of models](https://mlsanity.com/paper/2605.26409/) (2026)

## Abstract (arXiv)

Recent advances in self-supervised learning and neural network scaling have enabled the creation of large models, known as foundation models, which can be easily adapted to a wide range of downstream tasks. The current paradigm for comparing foundation models involves evaluating them with aggregate metrics on various benchmark datasets. This method of model comparison is heavily dependent on the chosen evaluation metric, which makes it unsuitable for situations where the ideal metric is either not obvious or unavailable. In this work, we present a methodology for directly comparing the embedding space geometry of foundation models, which facilitates model comparison without the need for an explicit evaluation metric. Our methodology is grounded in random graph theory and enables valid hypothesis testing of embedding similarity on a per-datum basis. Further, we demonstrate how our methodology can be extended to facilitate population level model comparison. In particular, we show how our framework can induce a manifold of models equipped with a distance function that correlates strongly with several downstream metrics. We remark on the utility of this population level model comparison as a first step towards a taxonomic science of foundation models.