mlsanityData to 14 Oct 2026

Big Transfer (BiT): General Visual Representation Learning

arXiv 1912.11370 ↗First posted 24 Dec 2019cs.CV cs.LGcited by 769

Not on the map. Why?

Cited by 22 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.

Key facts. Cited by 769 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 201; 2022: 199; 2023: 164; 2024: 106; 2025: 66; 2026: 33.

769 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2022A ConvNet for the 2020scited by 3,579
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2021Vision Transformers for Dense Predictioncited by 1,803
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  12. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
38 more
  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  2. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  3. 2021Scaling Vision Transformerscited by 999
  4. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  5. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  6. 2021Robust fine-tuning of zero-shot modelscited by 797
  7. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  10. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  11. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 642
  12. 2021ImageNet-21K Pretraining for the Massescited by 613
  13. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  14. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  15. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  16. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  17. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  18. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  19. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  20. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  21. 2022DeiT III: Revenge of the ViTcited by 419
  22. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  23. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 405
  24. 2021Early Convolutions Help Transformers See Bettercited by 396
  25. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  26. 2022Kubric: A scalable dataset generatorcited by 381
  27. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  28. 2021Explaining Neural Scaling Lawscited by 363
  29. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 339
  30. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
  31. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
  32. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  33. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  34. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  35. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  36. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  37. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  38. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259

50 most cited of 769 All 769 (JSON)