# A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark

arXiv: https://arxiv.org/abs/1910.04867 · HTML: https://mlsanity.com/paper/1910.04867/

## Key facts

- Cited by 425 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2019.

## Built on by (most cited first, 30 of 425)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning](https://mlsanity.com/paper/2210.08823/) (2022)
- [Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need](https://mlsanity.com/paper/2303.07338/) (2023)
- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [LogME: Practical Assessment of Pre-trained Models for Transfer Learning](https://mlsanity.com/paper/2102.11005/) (2021)
- [Benchmarking Representation Learning for Natural World Image Collections](https://mlsanity.com/paper/2103.16483/) (2021)
- [CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training](https://mlsanity.com/paper/2210.01055/) (2022)
- [Continual Learning with Pre-Trained Models: A Survey](https://mlsanity.com/paper/2401.16386/) (2024)
- [Finetune like you pretrain: Improved finetuning of zero-shot vision models](https://mlsanity.com/paper/2212.00638/) (2022)
- [ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models](https://mlsanity.com/paper/2204.08790/) (2022)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Hyperbolic Image-Text Representations](https://mlsanity.com/paper/2304.09172/) (2023)
- [Neural Prompt Search](https://mlsanity.com/paper/2206.04673/) (2022)
- [Revealing the Dark Secrets of Masked Image Modeling](https://mlsanity.com/paper/2205.13543/) (2022)
- [Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning](https://mlsanity.com/paper/2403.12030/) (2024)
- [Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization](https://mlsanity.com/paper/2311.06243/) (2023)

## Abstract (arXiv)

Representation learning promises to unlock deep learning for the long tail of vision tasks without expensive labelled datasets. Yet, the absence of a unified evaluation for general visual representations hinders progress. Popular protocols are often too constrained (linear classification), limited in diversity (ImageNet, CIFAR, Pascal-VOC), or only weakly related to representation quality (ELBO, reconstruction error). We present the Visual Task Adaptation Benchmark (VTAB), which defines good representations as those that adapt to diverse, unseen tasks with few examples. With VTAB, we conduct a large-scale study of many popular publicly-available representation learning algorithms. We carefully control confounders such as architecture and tuning budget. We address questions like: How effective are ImageNet representations beyond standard natural datasets? How do representations trained via generative and discriminative models compare? To what extent can self-supervision replace labels? And, how close are we to general visual representations?