mlsanityData to 14 Oct 2026

CMA-ES for Hyperparameter Optimization of Deep Neural Networks

arXiv 1604.07269 ↗First posted 25 Apr 2016cs.NE cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions. We provide a toy example comparing CMA-ES and state-of-the-art Bayesian optimization algorithms for tuning the hyperparameters of a convolutional neural network for the MNIST dataset on 30 GPUs in parallel.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). First posted on arXiv 25 Apr 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 5; 2023: 10; 2024: 5; 2025: 12; 2026: 4.

48 papers.

Sort:

By area:

  1. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 230
  2. 2021Automated Self-Supervised Learning for Graphscited by 50
  3. 2022Approximate Differentiable Rendering with Algebraic Surfacescited by 44
  4. 2023Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oraclescited by 35
  5. 2021Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimizationcited by 19
  6. 2021Fast and Informative Model Selection using Learning Curve Cross-Validationcited by 15
  7. 2021A Systematic Comparison Study on Hyperparameter Optimisation of Graph Neural Networks for Molecular Property Predictioncited by 9
  8. 2022c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimizationcited by 9
  9. 2023CMA-ES for Post Hoc Ensembling in AutoML: A Great Success and Salvageable Failurecited by 8
  10. 2025Reasoning BO: Enhancing Bayesian Optimization with Long-Context Reasoning Power of LLMscited by 8
  11. 2024Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimizationcited by 7
  12. 2021Learning Deep Morphological Networks with Neural Architecture Searchcited by 6
36 more
  1. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
  2. 2024High-dimensional Bayesian Optimization via Covariance Matrix Adaptation Strategycited by 6
  3. 2024Hyperparameter Optimization Can Even be Harmful in Off-Policy Learning and How to Deal with Itcited by 5
  4. 2021Hyperboost: Hyperparameter Optimization by Gradient Boosting surrogate modelscited by 4
  5. 2021Using Sequential Statistical Tests for Efficient Hyperparameter Tuningcited by 3
  6. 2026RoboPhD: Self-Improving Text-to-SQL Through Autonomous Agent Evolutioncited by 3
  7. 2021GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networkscited by 2
  8. 2021A Genetic Algorithm with Tree-structured Mutation for Hyperparameter Optimisation of Graph Neural Networkscited by 2
  9. 2022Using explainability to design physics-aware CNNs for solving subsurface inverse problemscited by 2
  10. 2023Big Dataunicodex2013Supply Chain Management Framework for Forecasting: Data Preprocessing and Machine Learning Techniquescited by 2
  11. 2023Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Searchcited by 2
  12. 2025Explicit and Non-asymptotic Query Complexities of Rank-Based Zeroth-order Algorithms on Smooth Functionscited by 2
  13. 2022Machine learning optimization of Majorana hybrid nanowirescited by 1
  14. 2023HomOpt: A Homotopy-Based Hyperparameter Optimization Methodcited by 1
  15. 2024Tensor Network Estimation of Distribution Algorithmscited by 1
  16. 2021Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloningcited by 0
  17. 2021Simple and Effective Balance of Contrastive Lossescited by 0
  18. 2022Hyperparameter Optimization with Neural Network Pruningcited by 0
  19. 2023Optimizing Neural Networks through Activation Function Discovery and Automatic Weight Initializationcited by 0
  20. 2023Topological gap protocol based machine learning optimization of Majorana hybrid wirescited by 0
  21. 2023Mirror Natural Evolution Strategiescited by 0
  22. 2023Asynchronous Evolution of Deep Neural Network Architecturescited by 0
  23. 2024MeGA: Merging Multiple Independently Trained Neural Networks Based on Genetic Algorithmcited by 0
  24. 2025Evolving Hard Maximum Cut Instances for Quantum Approximate Optimization Algorithmscited by 0
  25. 2025Unveiling and Steering Connectome Organization with Interpretable Latent Variablescited by 0
  26. 2025You Only Train Oncecited by 0
  27. 2025Tree-Structured Parzen Estimator Can Solve Black-Box Combinatorial Optimization More Efficientlycited by 0
  28. 2025Neural Architecture Search with Mixed Bio-inspired Learning Rulescited by 0
  29. 2025An Explainable, Attention-Enhanced, Bidirectional Long Short-Term Memory Neural Network for Joint 48-Hour Forecasting of Temperature, Irradiance, and Relative Humiditycited by 0
  30. 2025QIBONN: A Quantum-Inspired Bilevel Optimizer for Neural Networks on Tabular Classificationcited by 0
  31. 2025Batch Acquisition Function Evaluations and Decouple Optimizer Updates for Faster Bayesian Optimizationcited by 0
  32. 2025BEP: A Binary Error Propagation Algorithm for Binary Neural Networks Trainingcited by 0
  33. 2025Explicit and Non-asymptotic Query Complexities of Rank-Based Zeroth-order Algorithm on Stochastic Smooth Functionscited by 0
  34. 2026Evolution Strategy-Based Calibration for Low-Bit Quantization of Speech Modelscited by 0
  35. 2026StereoFactory: A Unified Merging Framework for Robust Stereo Matchingcited by 0
  36. 2026Adaptive Partitioning Schemes for Optimistic Optimizationcited by 0