Data to 7 Oct 2026

Lookahead Optimizer: k steps forward, 1 step back

arXiv 1907.08610 ↗First posted 19 Jul 2019cs.LG cs.NE stat.MLcited by 333

Not on the map. Why?

Abstract (arXiv)

The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categorized into two approaches: (1) adaptive learning rate schemes, such as AdaGrad and Adam, and (2) accelerated schemes, such as heavy-ball and Nesterov momentum. In this paper, we propose a new optimization algorithm, Lookahead, that is orthogonal to these previous approaches and iteratively updates two sets of weights. Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of fast weights generated by another optimizer. We show that Lookahead improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost. We empirically demonstrate Lookahead can significantly improve the performance of SGD and Adam, even with their default hyperparameter settings on ImageNet, CIFAR-10/100, neural machine translation, and Penn Treebank.

Key facts. Cited by 333 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 85; 2022: 61; 2023: 59; 2024: 46; 2025: 40; 2026: 42.

333 papers.

Sort:

By area:

2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
2021Robust fine-tuning of zero-shot modelscited by 785
2021TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classificationcited by 549
2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
2021Personalized Federated Learning using Hypernetworkscited by 210
2021High-Fidelity GAN Inversion for Image Attribute Editingcited by 197
2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
2024The Road Less Scheduledcited by 158
2021EditGAN: High-Precision Semantic Image Editingcited by 156
2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
2021GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimatorcited by 148

38 more

2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
2021Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizationcited by 141
2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 112
2021The Efficiency Misnomercited by 98
2022Particle Transformer for Jet Taggingcited by 95
2022GPV-Pose: Category-level Object Pose Estimation via Geometry-guided Point-wise Votingcited by 94
2021Only a Matter of Style: Age Transformation Using a Style-Based Regression Modelcited by 92
2023Deep Model Fusion: A Surveycited by 89
2021SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimationcited by 88
2021HyperInverter: Improving StyleGAN Inversion via Hypernetworkcited by 85
2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
2023β-Variational autoencoders and transformers for reduced-order modelling of fluid flowscited by 76
2022CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classificationcited by 65
2021OmniDet: Surround View Cameras based Multi-task Visual Perception Network for Autonomous Drivingcited by 57
2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
2023HS-Pose: Hybrid Scope Feature Extraction for Category-level Object Pose Estimationcited by 47
2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
2022Differentiable Zooming for Multiple Instance Learning on Whole-Slide Imagescited by 43
2022RBP-Pose: Residual Bounding Box Projection for Category-Level Pose Estimationcited by 43
2024Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancementcited by 43
2022SSP-Pose: Symmetry-Aware Shape Prior Deformation for Direct Category-Level Object Pose Estimationcited by 41
2023TriPlaneNet: An Encoder for EG3D Inversioncited by 41
2023Structured State Space Models for Multiple Instance Learning in Digital Pathologycited by 41
2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
2022SAM as an Optimal Relaxation of Bayescited by 40
2021Self-supervised driven consistency training for annotation efficient histopathology image analysiscited by 39
2021PingAn-VCGroup's Solution for ICDAR 2021 Competition on Scientific Literature Parsing Task B: Table Recognition to HTMLcited by 39
2021DualNet: Continual Learning, Fast and Slowcited by 39
2022Style Transformer for Image Inversion and Editingcited by 37
2023Make Encoder Great Again in 3D GAN Inversion through Geometry and Occlusion-Aware Encodingcited by 35
2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 35
2022Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimationcited by 34
2023Picture that Sketch: Photorealistic Image Generation from Abstract Sketchescited by 34
2021E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversioncited by 33
2021Physics-informed neural networks for the shallow-water equations on the spherecited by 32

50 most cited of 333 All 333 (JSON)