mlsanityData to 7 Oct 2026

Bayesian Optimization Meets Self-Distillation

arXiv 2304.12666 ↗First posted 25 Apr 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Bayesian optimization (BO) has contributed greatly to improving model performance by suggesting promising hyperparameter configurations iteratively based on observations from multiple training trials. However, only partial knowledge (i.e., the measured performances of trained models and their hyperparameter configurations) from previous trials is transferred. On the other hand, Self-Distillation (SD) only transfers partial knowledge learned by the task model itself. To fully leverage the various knowledge gained from all training trials, we propose the BOSS framework, which combines BO and SD. BOSS suggests promising hyperparameter configurations through BO and carefully selects pre-trained models from previous trials for SD, which are otherwise abandoned in the conventional BO process. BOSS achieves significantly better performance than both BO and SD in a wide range of tasks including general image classification, learning with noisy labels, semi-supervised learning, and medical image analysis tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 54 references that are in the map. First posted on arXiv 25 Apr 2023.

Builds on

33 of 54 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  8. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  9. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  10. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
21 more
  1. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  2. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  3. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  4. 2018Born Again Neural Networkscited by 539
  5. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  6. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  7. 2018HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Imagescited by 387
  8. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  9. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  10. 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144
  11. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  12. 2017Data Distillation: Towards Omni-Supervised Learningcited by 126
  13. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  14. 2018There Are Many Consistent Explanations of Unlabeled Data: Why You Should Averagecited by 82
  15. 2018Methods for Segmentation and Classification of Digital Microscopy Tissue Imagescited by 72
  16. 2019SRM : A Style-based Recalibration Module for Convolutional Neural Networkscited by 65
  17. 2018Bayesian Optimization in AlphaGocited by 55
  18. 2022Revisiting Self-Distillationcited by 23
  19. 2022Knowledge Distillation: Bad Models Can Be Good Role Modelscited by 12
  20. 2020Bayesian Optimization with a Prior for the Optimumcited by 5
  21. 2022Improving Multi-fidelity Optimization with a Recurring Learning Rate for Hyperparameter Tuningcited by 1

Built on by

Not cited yet.