# Towards a Principled Learning Rate Adaptation for Natural Evolution Strategies

arXiv: https://arxiv.org/abs/2112.10680 · HTML: https://mlsanity.com/paper/2112.10680/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 22 Nov 2021.

## Builds on (most cited first, 5 of 5)

- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Black-box optimization using geodesics in statistical manifolds](https://mlsanity.com/paper/1309.7168/) (2013)
- [Information-geometric optimization with natural selection](https://mlsanity.com/paper/1912.03395/) (2019)
- [Natural Evolution Strategy for Unconstrained and Implicitly Constrained Problems with Ridge Structure](https://mlsanity.com/paper/2108.09455/) (2021)
- [Maximum Likelihood-based Online Adaptation of Hyper-parameters in CMA-ES](https://mlsanity.com/paper/1406.2623/) (2014)

## Built on by (most cited first, 1 of 1)

- [Fast Moving Natural Evolution Strategy for High-Dimensional Problems](https://mlsanity.com/paper/2201.11422/) (2022)

## Abstract (arXiv)

Natural Evolution Strategies (NES) is a promising framework for black-box continuous optimization problems. NES optimizes the parameters of a probability distribution based on the estimated natural gradient, and one of the key parameters affecting the performance is the learning rate. We argue that from the viewpoint of the natural gradient method, the learning rate should be determined according to the estimation accuracy of the natural gradient. To do so, we propose a new learning rate adaptation mechanism for NES. The proposed mechanism makes it possible to set a high learning rate for problems that are relatively easy to optimize, which results in speeding up the search. On the other hand, in problems that are difficult to optimize (e.g., multimodal functions), the proposed mechanism makes it possible to set a conservative learning rate when the estimation accuracy of the natural gradient seems to be low, which results in the robust and stable search. The experimental evaluations on unimodal and multimodal functions demonstrate that the proposed mechanism works properly depending on a search situation and is effective over the existing method, i.e., using the fixed learning rate.