# Evolving Continuous Optimisers from Scratch

arXiv: https://arxiv.org/abs/2103.11746 · HTML: https://mlsanity.com/paper/2103.11746/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 54 references that are in the map.
- First posted on arXiv 22 Mar 2021.

## Builds on (most cited first, 9 of 9)

- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [AutoML-Zero: Evolving Machine Learning Algorithms From Scratch](https://mlsanity.com/paper/2003.03384/) (2020)
- [Learned Optimizers that Scale and Generalize](https://mlsanity.com/paper/1703.04813/) (2017)
- [Evolving the Structure of Evolution Strategies](https://mlsanity.com/paper/1610.05231/) (2016)
- [A new Taxonomy of Continuous Global Optimization Algorithms](https://mlsanity.com/paper/1808.08818/) (2018)
- [Mitigating Metaphors: A Comprehensible Guide to Recent Nature-Inspired Algorithms](https://mlsanity.com/paper/1902.08001/) (2019)
- [Instruction-Level Design of Local Optimisers using Push GP](https://mlsanity.com/paper/1905.10245/) (2019)
- [Optimising Optimisers with Push GP](https://mlsanity.com/paper/1910.00945/) (2019)

## Built on by (most cited first, 3 of 3)

- [Automated Design of Metaheuristic Algorithms: A Survey](https://mlsanity.com/paper/2303.06532/) (2023)
- [LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression](https://mlsanity.com/paper/2505.18602/) (2025)
- [AutoOpt: A General Framework for Automatically Designing Metaheuristic Optimization Algorithms with Diverse Structures](https://mlsanity.com/paper/2204.00998/) (2022)

## Abstract (arXiv)

This work uses genetic programming to explore the space of continuous optimisers, with the goal of discovering novel ways of doing optimisation. In order to keep the search space broad, the optimisers are evolved from scratch using Push, a Turing-complete, general-purpose, language. The resulting optimisers are found to be diverse, and explore their optimisation landscapes using a variety of interesting, and sometimes unusual, strategies. Significantly, when applied to problems that were not seen during training, many of the evolved optimisers generalise well, and often outperform existing optimisers. This supports the idea that novel and effective forms of optimisation can be discovered in an automated manner. This paper also shows that pools of evolved optimisers can be hybridised to further increase their generality, leading to optimisers that perform robustly over a broad variety of problem types and sizes.