mlsanityData to 7 Oct 2026

Blind Knowledge Distillation for Robust Image Classification

arXiv 2211.11355 ↗First posted 21 Nov 2022cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Optimizing neural networks with noisy labels is a challenging task, especially if the label set contains real-world noise. Networks tend to generalize to reasonable patterns in the early training stages and overfit to specific details of noisy samples in the latter ones. We introduce Blind Knowledge Distillation - a novel teacher-student approach for learning with noisy labels by masking the ground truth related teacher output to filter out potentially corrupted knowledge and to estimate the tipping point from generalizing to overfitting. Based on this, we enable the estimation of noise in the training data with Otsus algorithm. With this estimation, we train the network with a modified weighted cross-entropy loss function. We show in our experiments that Blind Knowledge Distillation detects overfitting effectively during training and improves the detection of clean and noisy labels on the recently published CIFAR-N dataset. Code is available at GitHub.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

18 of 23 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2020Knowledge Distillation: A Surveycited by 1,578
  5. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  6. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  7. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  8. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
  9. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  10. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  11. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
  12. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
6 more
  1. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 253
  2. 2020Learning with Instance-Dependent Label Noise: A Sample Sieve Approachcited by 158
  3. 2019Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Ratescited by 142
  4. 2021Understanding and Improving Early Stopping for Learning with Noisy Labelscited by 112
  5. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  6. 2020A Second-Order Approach to Learning with Instance-Dependent Label Noisecited by 90

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 1; 2025: 2.

9 papers.

  1. 2023HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularizationcited by 9
  2. 2023DSD2: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?cited by 6
  3. 2023Compensation Learning in Semantic Segmentationcited by 5
  4. 2025UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Modelcited by 4
  5. 2024ANNE: Adaptive Nearest Neighbors and Eigenvector-based Sample Selection for Robust Learning with Noisy Labelscited by 3
  6. 2023PASS: Peer-Agreement based Sample Selection for training with Noisy Labelscited by 1
  7. 2023Bridging Generative and Discriminative Noisy-Label Learning via Direction-Agnostic EM Formulationcited by 1
  8. 2023The Quest of Finding the Antidote to Sparse Double Descentcited by 0
  9. 2025Quantum-Inspired Spectral Geometry for Neural Operator Equivalence and Structured Pruningcited by 0