mlsanityData to 7 Oct 2026

A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark

arXiv 1910.04867 ↗First posted 1 Oct 2019cs.CV cs.LG stat.MLcited by 419

Not on the map. Why?

Cited by 27 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Representation learning promises to unlock deep learning for the long tail of vision tasks without expensive labelled datasets. Yet, the absence of a unified evaluation for general visual representations hinders progress. Popular protocols are often too constrained (linear classification), limited in diversity (ImageNet, CIFAR, Pascal-VOC), or only weakly related to representation quality (ELBO, reconstruction error). We present the Visual Task Adaptation Benchmark (VTAB), which defines good representations as those that adapt to diverse, unseen tasks with few examples. With VTAB, we conduct a large-scale study of many popular publicly-available representation learning algorithms. We carefully control confounders such as architecture and tuning budget. We address questions like: How effective are ImageNet representations beyond standard natural datasets? How do representations trained via generative and discriminative models compare? To what extent can self-supervision replace labels? And, how close are we to general visual representations?

Key facts. Cited by 419 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 35; 2022: 50; 2023: 96; 2024: 111; 2025: 79; 2026: 48.

419 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2022Visual Prompt Tuningcited by 1,674
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2021Scaling Vision Transformerscited by 993
  8. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  11. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  12. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
38 more
  1. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  2. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  3. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  4. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  5. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  6. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  7. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  8. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  9. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  10. 2024Continual Learning with Pre-Trained Models: A Surveycited by 151
  11. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  12. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  13. 2021The Benchmark Lotterycited by 133
  14. 2022Neural Prompt Searchcited by 133
  15. 2023Hyperbolic Image-Text Representationscited by 132
  16. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  17. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  18. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  19. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  20. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  21. 2022USB: A Unified Semi-supervised Learning Benchmark for Classificationcited by 95
  22. 2021Learning to See by Looking at Noisecited by 92
  23. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  24. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  25. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 81
  26. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
  27. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  28. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  29. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
  30. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  31. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 74
  32. 2021TorchGeo: Deep Learning With Geospatial Datacited by 73
  33. 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
  34. 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
  35. 2022LPT: Long-tailed Prompt Tuning for Image Classificationcited by 69
  36. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 68
  37. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  38. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65

50 most cited of 419 All 419 (JSON)