# Meta Mirror Descent: Optimiser Learning for Fast Convergence

arXiv: https://arxiv.org/abs/2203.02711 · HTML: https://mlsanity.com/paper/2203.02711/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 5 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Forward and Reverse Gradient-Based Hyperparameter Optimization](https://mlsanity.com/paper/1703.01785/) (2017)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [Learned Optimizers that Scale and Generalize](https://mlsanity.com/paper/1703.04813/) (2017)
- [Meta-Learning with Warped Gradient Descent](https://mlsanity.com/paper/1909.00025/) (2019)
- [Neural Optimizer Search with Reinforcement Learning](https://mlsanity.com/paper/1709.07417/) (2017)
- [Meta-Curvature](https://mlsanity.com/paper/1902.03356/) (2019)
- [Cold Case: The Lost MNIST Digits](https://mlsanity.com/paper/1905.10498/) (2019)
- [Generalization bounds for deep convolutional neural networks](https://mlsanity.com/paper/1905.12600/) (2019)
- [Distance-Based Regularisation of Deep Networks for Fine-Tuning](https://mlsanity.com/paper/2002.08253/) (2020)
- [Searching for Robustness: Loss Learning for Noisy Classification Tasks](https://mlsanity.com/paper/2103.00243/) (2021)
- [Gradient-based Hyperparameter Optimization Over Long Horizons](https://mlsanity.com/paper/2007.07869/) (2020)
- [Meta-Learning Bidirectional Update Rules](https://mlsanity.com/paper/2104.04657/) (2021)

## Built on by (most cited first, 7 of 7)

- [A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergence](https://mlsanity.com/paper/2301.13139/) (2023)
- [MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters](https://mlsanity.com/paper/2402.02342/) (2024)
- [Optimization-Inspired Few-Shot Adaptation for Large Language Models](https://mlsanity.com/paper/2505.19107/) (2025)
- [Meta-Learning with Versatile Loss Geometries for Fast Adaptation Using Mirror Descent](https://mlsanity.com/paper/2312.13486/) (2023)
- [Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning](https://mlsanity.com/paper/2501.12115/) (2025)
- [Learnable Loss Geometries with Mirror Descent for Scalable and Convergent Meta-Learning](https://mlsanity.com/paper/2509.02418/) (2025)
- [Non-Linear Strategic Classification Made Practical](https://mlsanity.com/paper/2606.28204/) (2026)

## Abstract (arXiv)

Optimisers are an essential component for training machine learning models, and their design influences learning speed and generalisation. Several studies have attempted to learn more effective gradient-descent optimisers via solving a bi-level optimisation problem where generalisation error is minimised with respect to optimiser parameters. However, most existing optimiser learning methods are intuitively motivated, without clear theoretical support. We take a different perspective starting from mirror descent rather than gradient descent, and meta-learning the corresponding Bregman divergence. Within this paradigm, we formalise a novel meta-learning objective of minimising the regret bound of learning. The resulting framework, termed Meta Mirror Descent (MetaMD), learns to accelerate optimisation speed. Unlike many meta-learned optimisers, it also supports convergence and generalisation guarantees and uniquely does so without requiring validation data. We evaluate our framework on a variety of tasks and architectures in terms of convergence rate and generalisation error and demonstrate strong performance.