# Probing the State of the Art: A Critical Look at Visual Representation Evaluation

arXiv: https://arxiv.org/abs/1912.00215 · HTML: https://mlsanity.com/paper/1912.00215/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2019.

## Built on by (most cited first, 6 of 6)

- [Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles](https://mlsanity.com/paper/2103.05121/) (2021)
- [Evaluating Self-Supervised Learning for Molecular Graph Embeddings](https://mlsanity.com/paper/2206.08005/) (2022)
- [On the Origins of the Block Structure Phenomenon in Neural Network Representations](https://mlsanity.com/paper/2202.07184/) (2022)
- [Probing Graph Representations](https://mlsanity.com/paper/2303.03951/) (2023)
- [Evaluating Representations with Readout Model Switching](https://mlsanity.com/paper/2302.09579/) (2023)
- [Socially Supervised Representation Learning: the Role of Subjectivity in Learning Efficient Representations](https://mlsanity.com/paper/2109.09390/) (2021)

## Abstract (arXiv)

Self-supervised research improved greatly over the past half decade, with much of the growth being driven by objectives that are hard to quantitatively compare. These techniques include colorization, cyclical consistency, and noise-contrastive estimation from image patches. Consequently, the field has settled on a handful of measurements that depend on linear probes to adjudicate which approaches are the best. Our first contribution is to show that this test is insufficient and that models which perform poorly (strongly) on linear classification can perform strongly (weakly) on more involved tasks like temporal activity localization. Our second contribution is to analyze the capabilities of five different representations. And our third contribution is a much needed new dataset for temporal activity localization.