mlsanityData to 14 Oct 2026

Multilayer Lookahead: a Nested Version of Lookahead

arXiv 2110.14254 ↗First posted 27 Oct 2021cs.LG cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

In recent years, SGD and its variants have become the standard tool to train Deep Neural Networks. In this paper, we focus on the recently proposed variant Lookahead, which improves upon SGD in a wide range of applications. Following this success, we study an extension of this algorithm, the Multilayer Lookahead optimizer, which recursively wraps Lookahead around itself. We prove the convergence of Multilayer Lookahead with two layers to a stationary point of smooth non-convex functions with O(frac1√T) rate. We also justify the improved generalization of both Lookahead over SGD, and of Multilayer Lookahead over Lookahead, by showing how they amplify the implicit regularization effect of SGD. We empirically verify our results and show that Multilayer Lookahead outperforms Lookahead on CIFAR-10 and CIFAR-100 classification tasks, and on GANs training on the MNIST dataset.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

18 of 27 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Improved Techniques for Training GANscited by 2,970
  4. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  6. 2018On First-Order Meta-Learning Algorithmscited by 1,018
  7. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  8. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  9. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  10. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  11. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  12. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 199
6 more
  1. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  2. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  3. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  4. 2020Implicit Gradient Regularizationcited by 146
  5. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 48
  6. 2020Taming GANs with Lookahead-Minmaxcited by 16

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 1.

2 papers.

  1. 2023Stable Nonconvex-Nonconcave Training via Linear Interpolationcited by 3
  2. 2025Generalization and Optimization of SGD with Lookaheadcited by 0