mlsanityData to 7 Oct 2026

What Do Neural Networks Learn When Trained With Random Labels?

arXiv 2006.10455 ↗First posted 18 Jun 2020stat.ML cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

We study deep neural networks (DNNs) trained on natural image data with entirely random labels. Despite its popularity in the literature, where it is often used to study memorization, generalization, and other phenomena, little is known about what DNNs learn in this setting. In this paper, we show analytically for convolutional and fully connected networks that an alignment between the principal components of network parameters and data takes place when training with random labels. We study this alignment effect by investigating neural networks pre-trained on randomly labelled image data and subsequently fine-tuned on disjoint datasets with random or real labels. We show how this alignment produces a positive transfer: networks pre-trained with random labels train faster downstream compared to training from scratch even after accounting for simple effects, such as weight scaling. We analyze how competing effects, such as specialization at later layers, may hide the positive transfer. These effects are studied in several network architectures, including VGG16 and ResNet18, on CIFAR10 and ImageNet.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 17; 2023: 10; 2024: 7; 2025: 1; 2026: 4.

58 papers.

Sort:

By area:

  1. 2022A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learningcited by 152
  2. 2021FINE Samples for Learning with Noisy Labelscited by 82
  3. 2021BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Searchcited by 63
  4. 2021A Theory of PAC Learnability of Partial Concept Classescited by 59
  5. 2021Exploring Memorization in Adversarial Trainingcited by 55
  6. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  7. 2021What can linearized neural networks actually say about generalization?cited by 49
  8. 2023When Source-Free Domain Adaptation Meets Learning with Noisy Labelscited by 46
  9. 2021Rethinking Graph Auto-Encoder Models for Attributed Graph Clusteringcited by 37
  10. 2021Neural Architecture Search with Random Labelscited by 33
  11. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  12. 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
38 more
  1. 2022Gaussian Universality of Perceptrons with Random Labelscited by 26
  2. 2022Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learningcited by 25
  3. 2022On Learning Contrastive Representations for Learning with Noisy Labelscited by 21
  4. 2021The Impact of Reinitialization on Generalization in Convolutional Neural Networkscited by 20
  5. 2024Memorization in deep learning: A surveycited by 14
  6. 2021How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspectivecited by 13
  7. 2021When Deep Classifiers Agree: Analyzing Correlations between Learning Order and Image Statisticscited by 11
  8. 2022The Curious Case of Benign Memorizationcited by 11
  9. 2023Classification of Heavy-tailed Features in High Dimensions: a Superstatistical Approachcited by 11
  10. 2023When does Privileged Information Explain Away Label Noise?cited by 10
  11. 2022What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environmentcited by 9
  12. 2021Understanding Memorization from the Perspective of Optimization via Efficient Influence Estimationcited by 7
  13. 2022Latent Topology Induction for Understanding Contextualized Representationscited by 7
  14. 2023In Search of a Data Transformation That Accelerates Neural Field Trainingcited by 7
  15. 2021Representation Alignment in Neural Networkscited by 6
  16. 2022ReFine: Re-randomization before Fine-tuning for Cross-domain Few-shot Learningcited by 6
  17. 2021Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networkscited by 5
  18. 2021Compositional generalization in semantic parsing with pretrained transformerscited by 5
  19. 2022Degradation-agnostic Correspondence from Resolution-asymmetric Stereocited by 5
  20. 2022Label-only Model Inversion Attack: The Attack that Requires the Least Informationcited by 4
  21. 2023A Contrastive Variational Graph Auto-Encoder for Node Clusteringcited by 4
  22. 2024Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noisecited by 4
  23. 2024On the universality of neural encodings in CNNscited by 4
  24. 2021On the Impact of Stable Ranks in Deep Netscited by 3
  25. 2021Energy Attack: On Transferring Adversarial Examplescited by 2
  26. 2022Layer-Stack Temperature Scalingcited by 2
  27. 2022Leveraging Unlabeled Data to Track Memorizationcited by 2
  28. 2023Compiled Models, Built-In Exploits: Uncovering Pervasive Bit-Flip Attack Surfaces in DNN Executablescited by 2
  29. 2023Pi-DUAL: Using Privileged Information to Distinguish Clean from Noisy Labelscited by 2
  30. 2024Spring-block theory of feature learning in deep neural networkscited by 2
  31. 2021Regularization by Misclassification in ReLU Neural Networkscited by 1
  32. 2022What do CNNs Learn in the First Layer and Why? A Linear Systems Perspectivecited by 1
  33. 2024Partitioned Neural Network Training via Synthetic Intermediate Labelscited by 1
  34. 2026From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMscited by 1
  35. 2021Contrastive Representations for Label Noise Require Fine-Tuningcited by 0
  36. 2022UnrealNAS: Can We Search Neural Architectures with Unreal Data?cited by 0
  37. 2022On Robust Learning from Noisy Labels: A Permutation Layer Approachcited by 0
  38. 2023Unsupervised Learning of Initialization in Deep Neural Networks via Maximum Mean Discrepancycited by 0

50 most cited of 58 All 58 (JSON)