# Regret-Aware Black-Box Optimization with Natural Gradients, Trust-Regions and Entropy Control

arXiv: https://arxiv.org/abs/2206.06090 · HTML: https://mlsanity.com/paper/2206.06090/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 55 references that are in the map.
- First posted on arXiv 24 May 2022.

## Builds on (most cited first, 10 of 10)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting](https://mlsanity.com/paper/1603.08785/) (2016)
- [The Differentiable Cross-Entropy Method](https://mlsanity.com/paper/1909.12830/) (2019)
- [Differentiable Trust Region Layers for Deep Reinforcement Learning](https://mlsanity.com/paper/2101.09207/) (2021)
- [Black-Box Data-efficient Policy Search for Robotics](https://mlsanity.com/paper/1703.07261/) (2017)
- [Compatible Natural Gradient Policy Search](https://mlsanity.com/paper/1902.02823/) (2019)
- [Expected Information Maximization: Using the I-Projection for Mixture Density Estimation](https://mlsanity.com/paper/2001.08682/) (2020)
- [COCO: The Experimental Procedure](https://mlsanity.com/paper/1603.08776/) (2016)

## Built on by (most cited first, 1 of 1)

- [Information-Theoretic Trust Regions for Stochastic Gradient-Based Optimization](https://mlsanity.com/paper/2310.20574/) (2023)

## Abstract (arXiv)

Most successful stochastic black-box optimizers, such as CMA-ES, use rankings of the individual samples to obtain a new search distribution. Yet, the use of rankings also introduces several issues such as the underlying optimization objective is often unclear, i.e., we do not optimize the expected fitness. Further, while these algorithms typically produce a high-quality mean estimate of the search distribution, the produced samples can have poor quality as these algorithms are ignorant of the regret. Lastly, noisy fitness function evaluations may result in solutions that are highly sub-optimal on expectation. In contrast, stochastic optimizers that are motivated by policy gradients, such as the Model-based Relative Entropy Stochastic Search (MORE) algorithm, directly optimize the expected fitness function without the use of rankings. MORE can be derived by applying natural policy gradients and compatible function approximation, and is using information theoretic constraints to ensure the stability of the policy update. While MORE does not suffer from the listed limitations, it often cannot achieve state of the art performance in comparison to ranking based methods. We improve MORE by decoupling the update of the mean and covariance of the search distribution allowing for more aggressive updates on the mean while keeping the update on the covariance conservative, an improved entropy scheduling technique based on an evolution path which results in faster convergence and a simplified and more effective model learning approach in comparison to the original paper. We compare our algorithm to state of the art black-box optimization algorithms on standard optimization tasks as well as on episodic RL tasks in robotics where it is also crucial to have small regret. We obtain competitive results on benchmark functions and clearly outperform ranking-based methods in terms of regret on the RL tasks.