mlsanityData to 14 Oct 2026

Distilling the Unknown to Unveil Certainty

arXiv 2311.07975 ↗First posted 14 Nov 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Out-of-distribution (OOD) detection is critical for identifying test samples that deviate from in-distribution (ID) data, ensuring network robustness and reliability. This paper presents a flexible framework for OOD knowledge distillation that extracts OOD-sensitive information from a network to develop a binary classifier capable of distinguishing between ID and OOD samples in both scenarios, with and without access to training ID data. To accomplish this, we introduce Confidence Amendment (CA), an innovative methodology that transforms an OOD sample into an ID one while progressively amending prediction confidence derived from the network to enhance OOD sensitivity. This approach enables the simultaneous synthesis of both ID and OOD samples, each accompanied by an adjusted prediction confidence, thereby facilitating the training of a binary classifier sensitive to OOD. Theoretical analysis provides bounds on the generalization error of the binary classifier, demonstrating the pivotal role of confidence amendment in enhancing OOD sensitivity. Extensive experiments spanning various datasets and network architectures confirm the efficacy of the proposed method in detecting OOD samples.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 67 of its 99 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

67 of 99 references.

References · 67

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2017Squeeze-and-Excitation Networkscited by 4,789
  10. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  11. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  12. 2016Matching Networks for One Shot Learningcited by 2,255
38 more
  1. 2013Describing Textures in the Wildcited by 2,125
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  5. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,355
  6. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  7. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  8. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  9. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  10. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  11. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
  12. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  13. 2020Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Datacited by 408
  14. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 405
  15. 2017Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samplescited by 389
  16. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 353
  17. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 339
  18. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  19. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  20. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 268
  21. 2015Unifying distillation and privileged informationcited by 233
  22. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  23. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  24. 2021A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detectioncited by 207
  25. 2019Data-Free Learning of Student Networkscited by 204
  26. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 179
  27. 2015TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Trackingcited by 165
  28. 2021A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challengescited by 154
  29. 2021Adversarial Reciprocal Points Learning for Open Set Recognitioncited by 153
  30. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 148
  31. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  32. 2023In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluationcited by 134
  33. 2022How to Exploit Hyperspherical Embeddings for Out-of-Distribution Detection?cited by 128
  34. 2022Full-Spectrum Out-of-Distribution Detectioncited by 63
  35. 2018Knowledge Distillation with Adversarial Samples Supporting Decision Boundarycited by 49
  36. 2022Boosting Out-of-distribution Detection with Typical Featurescited by 48
  37. 2021Neural Mean Discrepancy for Efficient Out-of-Distribution Detectioncited by 42
  38. 2019Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural networkcited by 39

50 most cited of 67 All 67 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Optimizing Three Critical Factors for Practical and Effective OOD Detection Fine-Tuning
  2. 2024Taming False Positives in Out-of-Distribution Detection with Human Feedback
  3. 2024Enhancing OOD Detection Using Latent Diffusion
  4. 2025COOkeD: Ensemble-based OOD detection in the era of zero-shot CLIP
  5. 2026From Subtle to Significant: Prompt-Driven Self-Improving Optimization in Test-Time Graph OOD Detection
  6. 2024Out-of-Distribution Learning with Human Feedback
  7. 2023Class Relevance Learning For Out-of-distribution Detection
  8. 2025Retrieval-Augmented Prompt for OOD Detection