# CMA-ES for Hyperparameter Optimization of Deep Neural Networks

arXiv: https://arxiv.org/abs/1604.07269 · HTML: https://mlsanity.com/paper/1604.07269/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2016.

## Built on by (most cited first, 30 of 48)

- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Automated Self-Supervised Learning for Graphs](https://mlsanity.com/paper/2106.05470/) (2021)
- [Approximate Differentiable Rendering with Algebraic Surfaces](https://mlsanity.com/paper/2207.10606/) (2022)
- [Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles](https://mlsanity.com/paper/2303.03751/) (2023)
- [Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization](https://mlsanity.com/paper/2105.01015/) (2021)
- [Fast and Informative Model Selection using Learning Curve Cross-Validation](https://mlsanity.com/paper/2111.13914/) (2021)
- [A Systematic Comparison Study on Hyperparameter Optimisation of Graph Neural Networks for Molecular Property Prediction](https://mlsanity.com/paper/2102.04283/) (2021)
- [c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization](https://mlsanity.com/paper/2211.14411/) (2022)
- [CMA-ES for Post Hoc Ensembling in AutoML: A Great Success and Salvageable Failure](https://mlsanity.com/paper/2307.00286/) (2023)
- [Reasoning BO: Enhancing Bayesian Optimization with Long-Context Reasoning Power of LLMs](https://mlsanity.com/paper/2505.12833/) (2025)
- [Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization](https://mlsanity.com/paper/2405.15393/) (2024)
- [Learning Deep Morphological Networks with Neural Architecture Search](https://mlsanity.com/paper/2106.07714/) (2021)
- [Learning Diverse Risk Preferences in Population-based Self-play](https://mlsanity.com/paper/2305.11476/) (2023)
- [High-dimensional Bayesian Optimization via Covariance Matrix Adaptation Strategy](https://mlsanity.com/paper/2402.03104/) (2024)
- [Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with It](https://mlsanity.com/paper/2404.15084/) (2024)
- [Hyperboost: Hyperparameter Optimization by Gradient Boosting surrogate models](https://mlsanity.com/paper/2101.02289/) (2021)
- [Using Sequential Statistical Tests for Efficient Hyperparameter Tuning](https://mlsanity.com/paper/2112.12438/) (2021)
- [RoboPhD: Self-Improving Text-to-SQL Through Autonomous Agent Evolution](https://mlsanity.com/paper/2601.01126/) (2026)
- [GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks](https://mlsanity.com/paper/2102.09298/) (2021)
- [A Genetic Algorithm with Tree-structured Mutation for Hyperparameter Optimisation of Graph Neural Networks](https://mlsanity.com/paper/2102.11995/) (2021)
- [Using explainability to design physics-aware CNNs for solving subsurface inverse problems](https://mlsanity.com/paper/2211.08651/) (2022)
- [Big Dataunicodex2013Supply Chain Management Framework for Forecasting: Data Preprocessing and Machine Learning Techniques](https://mlsanity.com/paper/2307.12971/) (2023)
- [Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Search](https://mlsanity.com/paper/2307.15621/) (2023)
- [Explicit and Non-asymptotic Query Complexities of Rank-Based Zeroth-order Algorithms on Smooth Functions](https://mlsanity.com/paper/2512.16200/) (2025)
- [Machine learning optimization of Majorana hybrid nanowires](https://mlsanity.com/paper/2208.02182/) (2022)
- [HomOpt: A Homotopy-Based Hyperparameter Optimization Method](https://mlsanity.com/paper/2308.03317/) (2023)
- [Tensor Network Estimation of Distribution Algorithms](https://mlsanity.com/paper/2412.19780/) (2024)
- [Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloning](https://mlsanity.com/paper/2112.08094/) (2021)
- [Simple and Effective Balance of Contrastive Losses](https://mlsanity.com/paper/2112.11743/) (2021)
- [Hyperparameter Optimization with Neural Network Pruning](https://mlsanity.com/paper/2205.08695/) (2022)

## Abstract (arXiv)

Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions. We provide a toy example comparing CMA-ES and state-of-the-art Bayesian optimization algorithms for tuning the hyperparameters of a convolutional neural network for the MNIST dataset on 30 GPUs in parallel.