mlsanityData to 7 Oct 2026

The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Image

arXiv 2112.00725 ↗First posted 1 Dec 2021cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

What can neural networks learn about the visual world when provided with only a single image as input? While any image obviously cannot contain the multitudes of all existing objects, scenes and lighting conditions - within the space of all 256^(3x224x224) possible 224-sized square images, it might still provide a strong prior for natural images. To analyze this `augmented image prior' hypothesis, we develop a simple framework for training neural networks from scratch using a single image and augmentations using knowledge distillation from a supervised pretrained teacher. With this, we find the answer to the above question to be: `surprisingly, a lot'. In quantitative terms, we find accuracies of 94%/74% on CIFAR-10/100, 69% on ImageNet, and by extending this method to video and audio, 51% on Kinetics-400 and 84% on SpeechCommands. In extensive analyses spanning 13 datasets, we disentangle the effect of augmentations, choice of data and network architectures and also provide qualitative evaluations that include lucid `panda neurons' in networks that have never even seen one.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 57 of its 82 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

57 of 82 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  12. 2016Wide Residual Networkscited by 3,057
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  5. 2020Knowledge Distillation: A Surveycited by 1,578
  6. 2018Group Normalizationcited by 1,514
  7. 2019Contrastive Multiview Codingcited by 1,297
  8. 2017Deep Image Priorcited by 1,097
  9. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  10. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  11. 2021Going deeper with Image Transformerscited by 759
  12. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  13. 2019Contrastive Representation Distillationcited by 734
  14. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  15. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  16. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  17. 2018Born Again Neural Networkscited by 539
  18. 2019Similarity-Preserving Knowledge Distillationcited by 507
  19. 2013Do Deep Nets Really Need to be Deep?cited by 466
  20. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  21. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  22. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  23. 2018Knockoff Nets: Stealing Functionality of Black-Box Modelscited by 285
  24. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  25. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  26. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  27. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  28. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  29. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  30. 2019Data-Free Learning of Student Networkscited by 203
  31. 2017Learning from Noisy Labels with Distillationcited by 160
  32. 2017Sobolev Training for Neural Networkscited by 136
  33. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  34. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
  35. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  36. 2017Data Distillation: Towards Omni-Supervised Learningcited by 126
  37. 2021Pre-training without Natural Imagescited by 100
  38. 2021Learning to See by Looking at Noisecited by 92

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 0; 2026: 1.

7 papers.

  1. 2023Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?cited by 21
  2. 2023SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Datacited by 9
  3. 2024Accessing Vision Foundation Models via ImageNet-1Kcited by 9
  4. 2024What Makes a Good Dataset for Knowledge Distillation?cited by 8
  5. 2024Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillationcited by 4
  6. 2024Federated Learning with a Single Shared Imagecited by 1
  7. 2026Soft Label Pruning and Quantization for Large-Scale Dataset Distillationcited by 0