mlsanityData to 7 Oct 2026

Knowledge Distillation with Adversarial Samples Supporting Decision Boundary

arXiv 1805.05532 ↗First posted 15 May 2018cs.LG cs.CV stat.MLcited by 49

Not on the map. Why?

Abstract (arXiv)

Many recent works on knowledge distillation have provided ways to transfer the knowledge of a trained network for improving the learning process of a new one, but finding a good technique for knowledge distillation is still an open problem. In this paper, we provide a new perspective based on a decision boundary, which is one of the most important component of a classifier. The generalization performance of a classifier is closely related to the adequacy of its decision boundary, so a good classifier bears a good decision boundary. Therefore, transferring information closely related to the decision boundary can be a good attempt for knowledge distillation. To realize this goal, we utilize an adversarial attack to discover samples supporting a decision boundary. Based on this idea, to transfer more accurate information about the decision boundary, the proposed algorithm trains a student classifier based on the adversarial samples supporting the decision boundary. Experiments show that the proposed method indeed improves knowledge distillation and achieves the state-of-the-arts performance.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 12; 2023: 15; 2024: 6; 2025: 4; 2026: 1.

49 papers.

Sort:

By area:

  1. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  2. 20213D-to-2D Distillation for Indoor Scene Parsingcited by 47
  3. 2021Co-advise: Cross Inductive Bias Distillationcited by 40
  4. 2021Qimera: Data-free Quantization with Synthetic Boundary Supporting Samplescited by 39
  5. 2023Lion: Adversarial Distillation of Proprietary Large Language Modelscited by 33
  6. 2023A comprehensive survey on deep active learning in medical image analysiscited by 32
  7. 2023Delving into the Adversarial Robustness of Federated Learningcited by 19
  8. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  9. 2022Self-Knowledge Distillation via Dropoutcited by 18
  10. 2021Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesiscited by 14
  11. 2021Convolutional Neural Network Compression through Generalized Kronecker Product Decompositioncited by 14
  12. 2022NeRN -- Learning Neural Representations for Neural Networkscited by 14
37 more
  1. 2021Thief, Beware of What Get You There: Towards Understanding Model Extraction Attackcited by 9
  2. 2023Cross-Class Feature Augmentation for Class Incremental Learningcited by 8
  3. 2024MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shiftscited by 8
  4. 2024What Makes a Good Dataset for Knowledge Distillation?cited by 8
  5. 2023Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillationcited by 7
  6. 2023One-Shot Federated Learning for LEO Constellations that Reduces Convergence Time from Days to 90 Minutescited by 7
  7. 2024Rethinking Impersonation and Dodging Attacks on Face Recognition Systemscited by 7
  8. 2022Adversarial Examples for Good: Adversarial Examples Guided Imbalanced Learningcited by 6
  9. 2023Illumination Distillation Framework for Nighttime Person Re-Identification and A New Benchmarkcited by 6
  10. 2022Role of Data Augmentation Strategies in Knowledge Distillation for Wearable Sensor Datacited by 5
  11. 2023Knowledge-guided Causal Intervention for Weakly-supervised Object Localizationcited by 4
  12. 2023GRAN: Ghost Residual Attention Network for Single Image Super Resolutioncited by 3
  13. 2022Holistic Approach to Measure Sample-level Adversarial Vulnerability and its Utility in Building Trustworthy Systemscited by 2
  14. 2023ACQ: Improving Generative Data-free Quantization Via Attention Correctioncited by 2
  15. 2023Evasion Adversarial Attacks Remain Impractical Against ML-based Network Intrusion Detection Systems, Especially Dynamic Onescited by 2
  16. 2024Finding Patterns in Ambiguity: Interpretable Stress Testing in the Decision~Boundarycited by 2
  17. 2025Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMscited by 2
  18. 2021Confidence Conditioned Knowledge Distillationcited by 1
  19. 2021Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Bettercited by 1
  20. 2022CEKD:Cross Ensemble Knowledge Distillation for Augmented Fine-grained Datacited by 1
  21. 2022Dive into Big Model Trainingcited by 1
  22. 2022Teacher Guided Training: An Efficient Framework for Knowledge Transfercited by 1
  23. 2025Guidelines For The Choice Of The Baseline in XAI Attribution Methodscited by 1
  24. 2021Distilling Knowledge via Intermediate Classifierscited by 0
  25. 2021BOSS: Bidirectional One-Shot Synthesis of Adversarial Examplescited by 0
  26. 2022Exact Feature Collisions in Neural Networkscited by 0
  27. 2022Rethinking Cost-sensitive Classification in Deep Learning via Adversarial Data Augmentationcited by 0
  28. 2022SeKron: A Decomposition Method Supporting Many Factorization Structurescited by 0
  29. 2022The Vanishing Decision Boundary Complexity and the Strong First Componentcited by 0
  30. 2023Improved knowledge distillation by utilizing backward pass knowledge in neural networkscited by 0
  31. 2023LIB-KD: Teaching Inductive Bias for Efficient Vision Transformer Distillation and Compressioncited by 0
  32. 2023Distilling the Unknown to Unveil Certaintycited by 0
  33. 2023Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalizationcited by 0
  34. 2024Imitating the Functionality of Image-to-Image Models Using a Single Examplecited by 0
  35. 2025Decision Boundary Optimization-Informed Domain Adaptationcited by 0
  36. 2025MMARD: Improving the Min-Max Optimization Process in Adversarial Robustness Distillationcited by 0
  37. 2026Smoothing the Black-Box: Signed-Distance Supervision for Black-Box Model Copyingcited by 0