Data to 7 Oct 2026

Born Again Neural Networks

arXiv 1805.04770 ↗First posted 12 May 2018stat.ML cs.AI cs.LGcited by 539

Not on the map. Why?

Abstract (arXiv)

Knowledge Distillation (KD) consists of transferring âknowledgeâ from one machine learning model (the teacher) to another (the student). Commonly, the teacher is a high-capacity model with formidable performance, while the student is more compact. By transferring knowledge, one hopes to benefit from the studentâs compactness, without sacrificing too much performance. We study KD from a new perspective: rather than compressing models, we train students parameterized identically to their teachers. Surprisingly, these Born-Again Networks (BANs), outperform their teachers significantly, both on computer vision and language modeling tasks. Our experiments with BANs based on DenseNets demonstrate state-of-the-art performance on the CIFAR-10 (3.5%) and CIFAR-100 (15.5%) datasets, by validation error. Additional experiments explore two distillation objectives: (i) Confidence-Weighted by Teacher Max (CWTM) and (ii) Dark Knowledge with Permuted Predictions (DKPP). Both methods elucidate the essential components of KD, demonstrating the effect of the teacher outputs on both predicted and non-predicted classes.

Key facts. Cited by 539 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 0; 2021: 130; 2022: 103; 2023: 94; 2024: 70; 2025: 59; 2026: 81.

539 papers.

Sort:

By area:

2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
2022Decoupled Knowledge Distillationcited by 380
2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
2023A Cookbook of Self-Supervised Learningcited by 248
2021R-Drop: Regularized Dropout for Neural Networkscited by 202
2021Does Knowledge Distillation Really Work?cited by 187
2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
2025Streaming 4D Visual Geometry Transformercited by 147
2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129

38 more

2022Class-Incremental Continual Learning into the eXtended DER-versecited by 121
2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 100
2022Membership Inference Attacks by Exploiting Loss Trajectorycited by 93
2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
2021SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentationcited by 86
2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 75
2021Self Supervision to Distillation for Long-Tailed Visual Recognitioncited by 73
2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
2024Robust Preference Optimization through Reward Model Distillationcited by 67
2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
2022Nested Collaborative Learning for Long-Tailed Visual Recognitioncited by 63
2021Annealing Knowledge Distillationcited by 61
2022Learning Generalizable Models for Vehicle Routing Problems via Knowledge Distillationcited by 61
2023Class Attention Transfer Based Knowledge Distillationcited by 61
2021Fair Feature Distillation for Visual Recognitioncited by 59
2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
2021Object DGCNN: 3D Object Detection using Dynamic Graphscited by 58
2023Decoupled Kullback-Leibler Divergence Losscited by 58
2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 58
2021Essentials for Class Incremental Learningcited by 57
2022L2G: A Simple Local-to-Global Knowledge Transfer Framework for Weakly Supervised Semantic Segmentationcited by 57
2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
2021Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learningcited by 56
2021Universal Representation Learning from Multiple Domains for Few-shot Classificationcited by 56
2021Localization Distillation for Dense Object Detectioncited by 55
2021Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!cited by 51
2021Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learningcited by 49
2021Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimationcited by 48
2022Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detectioncited by 48
2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46

50 most cited of 539 All 539 (JSON)