# Evaluating representations by the complexity of learning low-loss predictors

arXiv: https://arxiv.org/abs/2009.07368 · HTML: https://mlsanity.com/paper/2009.07368/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2020.

## Built on by (most cited first, 16 of 16)

- [FedAvg with Fine Tuning: Local Updates Lead to Representation Learning](https://mlsanity.com/paper/2205.13692/) (2022)
- [A Closer Look at How Fine-tuning Changes BERT](https://mlsanity.com/paper/2106.14282/) (2021)
- [The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color](https://mlsanity.com/paper/2110.08182/) (2021)
- [DirectProbe: Studying Representations without Classifiers](https://mlsanity.com/paper/2104.05904/) (2021)
- [DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability](https://mlsanity.com/paper/2210.00364/) (2022)
- [Rissanen Data Analysis: Examining Dataset Characteristics via Description Length](https://mlsanity.com/paper/2103.03872/) (2021)
- [A Bayesian Framework for Information-Theoretic Probing](https://mlsanity.com/paper/2109.03853/) (2021)
- [Sequential Learning Of Neural Networks for Prequential MDL](https://mlsanity.com/paper/2210.07931/) (2022)
- [Exploring the Latent Space of Autoencoders with Interventional Assays](https://mlsanity.com/paper/2106.16091/) (2021)
- [SynBench: Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data](https://mlsanity.com/paper/2210.02989/) (2022)
- [Evaluating Representations with Readout Model Switching](https://mlsanity.com/paper/2302.09579/) (2023)
- [Self-Calibrating BCIs: Ranking and Recovery of Mental Targets Without Labels](https://mlsanity.com/paper/2506.11151/) (2025)
- [Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data](https://mlsanity.com/paper/2607.11883/) (2026)
- [Socially Supervised Representation Learning: the Role of Subjectivity in Learning Efficient Representations](https://mlsanity.com/paper/2109.09390/) (2021)
- [Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods](https://mlsanity.com/paper/2206.01251/) (2022)
- [Excess Description Length of Learning Generalizable Predictors](https://mlsanity.com/paper/2601.04728/) (2026)

## Abstract (arXiv)

We consider the problem of evaluating representations of data for use in solving a downstream task. We propose to measure the quality of a representation by the complexity of learning a predictor on top of the representation that achieves low loss on a task of interest, and introduce two methods, surplus description length (SDL) and varepsilon sample complexity (varepsilonSC). In contrast to prior methods, which measure the amount of information about the optimal predictor that is present in a specific amount of data, our methods measure the amount of information needed from the data to recover an approximation of the optimal predictor up to a specified tolerance. We present a framework to compare these methods based on plotting the validation loss versus evaluation dataset size (the "loss-data" curve). Existing measures, such as mutual information and minimum description length probes, correspond to slices and integrals along the data axis of the loss-data curve, while ours correspond to slices and integrals along the loss axis. We provide experiments on real data to compare the behavior of each of these methods over datasets of varying size along with a high performance open source library for representation evaluation at https://github.com/willwhitney/reprieve.