mlsanityData to 14 Oct 2026

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

arXiv 1707.02968 ↗First posted 10 Jul 2017cs.CV cs.AIcited by 919

Not on the map. Why?

Abstract (arXiv)

The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10x or 100x? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between `enormous data' and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pre-training) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-the-art results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets.

Key facts. Cited by 919 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 228; 2022: 233; 2023: 195; 2024: 123; 2025: 96; 2026: 44.

919 papers.

Sort:

By area:

  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2021ViViT: A Video Vision Transformercited by 1,620
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  8. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,081
  9. 2021Scaling Vision Transformerscited by 999
  10. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  11. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  12. 2021Robust fine-tuning of zero-shot modelscited by 797
38 more
  1. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  3. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  4. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  5. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  6. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  7. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 615
  8. 2021ImageNet-21K Pretraining for the Massescited by 613
  9. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  10. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  11. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  12. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  13. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  14. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  15. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  16. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  17. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  18. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  19. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  20. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  21. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  22. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  23. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  24. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  25. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  26. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
  27. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  28. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  29. 2022How Do Vision Transformers Work?cited by 307
  30. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 291
  31. 2023Towards Generalist Biomedical AIcited by 289
  32. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  33. 2023Effective Data Augmentation With Diffusion Modelscited by 283
  34. 2022Matryoshka Representation Learningcited by 282
  35. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 264
  36. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  37. 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
  38. 2021Understanding Robustness of Transformers for Image Classificationcited by 249

50 most cited of 919 All 919 (JSON)