# Autotune: A Derivative-free Optimization Framework for Hyperparameter Tuning

arXiv: https://arxiv.org/abs/1804.07824 · HTML: https://mlsanity.com/paper/1804.07824/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2018.

## Built on by (most cited first, 20 of 20)

- [Bag of Tricks for Training Data Extraction from Language Models](https://mlsanity.com/paper/2302.04460/) (2023)
- [Transfer Learning based Search Space Design for Hyperparameter Tuning](https://mlsanity.com/paper/2206.02511/) (2022)
- [Towards Automated Imbalanced Learning with Deep Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2208.12433/) (2022)
- [OpenBox: A Python Toolkit for Generalized Black-box Optimization](https://mlsanity.com/paper/2304.13339/) (2023)
- [Minibatch Stochastic Three Points Method for Unconstrained Smooth Minimization](https://mlsanity.com/paper/2209.07883/) (2022)
- [PyHopper -- Hyperparameter optimization](https://mlsanity.com/paper/2210.04728/) (2022)
- [JITuNE: Just-In-Time Hyperparameter Tuning for Network Embedding Algorithms](https://mlsanity.com/paper/2101.06427/) (2021)
- [ChaCha for Online AutoML](https://mlsanity.com/paper/2106.04815/) (2021)
- [Hyper-parameter Tuning for Fair Classification without Sensitive Attribute Access](https://mlsanity.com/paper/2302.01385/) (2023)
- [Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization](https://mlsanity.com/paper/2303.03394/) (2023)
- [Generalizing Adam to Manifolds for Efficiently Training Transformers](https://mlsanity.com/paper/2305.16901/) (2023)
- [Hierarchical Collaborative Hyper-parameter Tuning](https://mlsanity.com/paper/2205.05272/) (2022)
- [SigOpt Mulch: An Intelligent System for AutoML of Gradient Boosted Trees](https://mlsanity.com/paper/2307.04849/) (2023)
- [Compact Model Parameter Extraction via Derivative-Free Optimization](https://mlsanity.com/paper/2406.16355/) (2024)
- [Stochastic Optimization with Random Search](https://mlsanity.com/paper/2510.15610/) (2025)
- [Learning Sampling Policy for Faster Derivative Free Optimization](https://mlsanity.com/paper/2104.04405/) (2021)
- [Hybrid Algorithm Selection and Hyperparameter Tuning on Distributed Machine Learning Resources: A Hierarchical Agent-based Approach](https://mlsanity.com/paper/2309.06604/) (2023)
- [Breaking MLPerf Training: A Case Study on Optimizing BERT](https://mlsanity.com/paper/2402.02447/) (2024)
- [Network architecture search of X-ray based scientific applications](https://mlsanity.com/paper/2404.10689/) (2024)
- [ReVeal-MT: A Physics-Informed Neural Network for Multi-Transmitter Radio Environment Mapping](https://mlsanity.com/paper/2512.04100/) (2025)

## Abstract (arXiv)

Machine learning applications often require hyperparameter tuning. The hyperparameters usually drive both the efficiency of the model training process and the resulting model quality. For hyperparameter tuning, machine learning algorithms are complex black-boxes. This creates a class of challenging optimization problems, whose objective functions tend to be nonsmooth, discontinuous, unpredictably varying in computational expense, and include continuous, categorical, and/or integer variables. Further, function evaluations can fail for a variety of reasons including numerical difficulties or hardware failures. Additionally, not all hyperparameter value combinations are compatible, which creates so called hidden constraints. Robust and efficient optimization algorithms are needed for hyperparameter tuning. In this paper we present an automated parallel derivative-free optimization framework called Autotune, which combines a number of specialized sampling and search methods that are very effective in tuning machine learning models despite these challenges. Autotune provides significantly improved models over using default hyperparameter settings with minimal user interaction on real-world applications. Given the inherent expense of training numerous candidate models, we demonstrate the effectiveness of Autotune's search methods and the efficient distributed and parallel paradigms for training and tuning models, and also discuss the resource trade-offs associated with the ability to both distribute the training process and parallelize the tuning process.