# Bayesian Optimization for Hyperparameters Tuning in Neural Networks

arXiv: https://arxiv.org/abs/2410.21886 · HTML: https://mlsanity.com/paper/2410.21886/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 7 of 7)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Constrained Efficient Global Optimization of Expensive Black-box Functions](https://mlsanity.com/paper/2211.00162/) (2022)
- [Loss Functions and Metrics in Deep Learning](https://mlsanity.com/paper/2307.02694/) (2023)
- [BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization](https://mlsanity.com/paper/1910.06403/) (2019)
- [VoteLab: A Modular and Adaptive Experimentation Platform for Online Collective Decision Making](https://mlsanity.com/paper/2307.10903/) (2023)

## Built on by (most cited first, 2 of 2)

- [ABG-NAS: Adaptive Bayesian Genetic Neural Architecture Search for Graph Representation Learning](https://mlsanity.com/paper/2504.21254/) (2025)
- [S2ML: Spatio-Spectral Mutual Learning for Depth Completion](https://mlsanity.com/paper/2511.06033/) (2025)

## Abstract (arXiv)

This study investigates the application of Bayesian Optimization (BO) for the hyperparameter tuning of neural networks, specifically targeting the enhancement of Convolutional Neural Networks (CNN) for image classification tasks. Bayesian Optimization is a derivative-free global optimization method suitable for expensive black-box functions with continuous inputs and limited evaluation budgets. The BO algorithm leverages Gaussian Process regression and acquisition functions like Upper Confidence Bound (UCB) and Expected Improvement (EI) to identify optimal configurations effectively. Using the Ax and BOTorch frameworks, this work demonstrates the efficiency of BO in reducing the number of hyperparameter tuning trials while achieving competitive model performance. Experimental outcomes reveal that BO effectively balances exploration and exploitation, converging rapidly towards optimal settings for CNN architectures. This approach underlines the potential of BO in automating neural network tuning, contributing to improved accuracy and computational efficiency in machine learning pipelines.