# Using learned optimizers to make models robust to input noise

arXiv: https://arxiv.org/abs/1906.03367 · HTML: https://mlsanity.com/paper/1906.03367/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2019.

## Built on by (most cited first, 10 of 10)

- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [Practical tradeoffs between memory, compute, and performance in learned optimizers](https://mlsanity.com/paper/2203.11860/) (2022)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [Training Learned Optimizers with Randomly Initialized Learned Optimizers](https://mlsanity.com/paper/2101.07367/) (2021)
- [Learn2Hop: Learned Optimization on Rough Landscapes](https://mlsanity.com/paper/2107.09661/) (2021)
- [Neural Fixed-Point Acceleration for Convex Optimization](https://mlsanity.com/paper/2107.10254/) (2021)
- [Mnemosyne: Learning to Train Transformers with Transformers](https://mlsanity.com/paper/2302.01128/) (2023)
- [Enhancing object detection robustness: A synthetic and natural perturbation approach](https://mlsanity.com/paper/2304.10622/) (2023)

## Abstract (arXiv)

State-of-the art vision models can achieve superhuman performance on image classification tasks when testing and training data come from the same distribution. However, when models are tested on corrupted images (e.g. due to scale changes, translations, or shifts in brightness or contrast), performance degrades significantly. Here, we explore the possibility of meta-training a learned optimizer that can train image classification models such that they are robust to common image corruptions. Specifically, we are interested training models that are more robust to noise distributions not present in the training data. We find that a learned optimizer meta-trained to produce models which are robust to Gaussian noise trains models that are more robust to Gaussian noise at other scales compared to traditional optimizers like Adam. The effect of meta-training is more complicated when targeting a more general set of noise distributions, but led to improved performance on half of held-out corruption tasks. Our results suggest that meta-learning provides a novel approach for studying and improving the robustness of deep learning models.