mlsanityData to 7 Oct 2026

SGDR: Stochastic Gradient Descent with Warm Restarts

arXiv 1608.03983 ↗First posted 13 Aug 2016cs.LG cs.NE math.OCcited by 4,993

Not on the map. Why?

Cited by 139 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR

Key facts. Cited by 4,993 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Aug 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 593; 2022: 833; 2023: 951; 2024: 1031; 2025: 902; 2026: 683.

4,993 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  6. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  10. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
38 more
  1. 2021Multiscale Vision Transformerscited by 896
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  3. 2021Membership Inference Attacks From First Principlescited by 839
  4. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  5. 2021Multi-Stage Progressive Image Restorationcited by 813
  6. 2021Robust fine-tuning of zero-shot modelscited by 785
  7. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  8. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  9. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  10. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  11. 2022Simple Baselines for Image Restorationcited by 701
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  13. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  14. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  15. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  16. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  17. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  18. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  19. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  20. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  21. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  22. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  23. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  24. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  25. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  26. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  27. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 537
  28. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  29. 2021CutPaste: Self-Supervised Learning for Anomaly Detection and Localizationcited by 529
  30. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  31. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  32. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  33. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  34. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  35. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  36. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  37. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  38. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456

50 most cited of 4,993 All 4,993 (JSON)