Diverse Feature Learning by Self-distillation and Reset
Not on the map. Why?
Abstract (arXiv)
Our paper addresses the problem of models struggling to learn diverse features, due to either forgetting previously learned features or failing to learn new ones. To overcome this problem, we introduce Diverse Feature Learning (DFL), a method that combines an important feature preservation algorithm with a new feature learning algorithm. Specifically, for preserving important features, we utilize self-distillation in ensemble models by selecting the meaningful model weights observed during training. For learning new features, we employ reset that involves periodically re-initializing part of the model. As a result, through experiments with various models on the image classification, we have identified the potential for synergistic effects between self-distillation and reset.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 29 Mar 2024.
Builds on
33 of 41 references.
References · 33
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2014Going Deeper with Convolutionscited by 4,231
- 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
- 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
- 2018Exploration by Random Network Distillationcited by 966
- 2022Continual Test-Time Domain Adaptationcited by 520
- 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
- 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
- 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
21 more
- 2022Perception Prioritized Training of Diffusion Modelscited by 301
- 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
- 2021Ensemble deep learning: A reviewcited by 283
- 2021Projected GANs Converge Fastercited by 204
- 2023Continual Pre-training of Language Modelscited by 194
- 2020Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformationcited by 183
- 2019Self-Ensembling with GAN-based Data Augmentation for Domain Adaptation in Semantic Segmentationcited by 126
- 2022Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning Systemcited by 110
- 2020Self-Knowledge Distillation with Progressive Refinement of Targetscited by 106
- 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
- 2022Learning Debiased Classifier with Biased Committeecited by 51
- 2023Analyzing Bias in Diffusion-based Face Generation Modelscited by 45
- 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
- 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
- 2022Fortuitous Forgetting in Connectionist Networkscited by 39
- 2022CAFA: Class-Aware Feature Alignment for Test-Time Adaptationcited by 21
- 2023ERM++: An Improved Baseline for Domain Generalizationcited by 18
- 2022Training Debiased Subnetworks with Contrastive Weight Pruningcited by 16
- 2022BiSTNet: Semantic Image Prior Guided Bidirectional Temporal Feature Fusion for Deep Exemplar-based Video Colorizationcited by 13
- 2022Deep Active Learning for Computer Vision: Past and Futurecited by 7
- 2023Improving Sketch Colorization using Adversarial Segmentation Consistencycited by 3
Built on by
Citing papers per year: 2024: 2.
2 papers.