mlsanityData to 7 Oct 2026

A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay

arXiv 1803.09820 ↗First posted 26 Mar 2018cs.LG cs.CV cs.NEcited by 223

Not on the map. Why?

Abstract (arXiv)

Although deep learning has produced dazzling successes for applications of image, speech, and video processing in the past few years, most trainings are with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyper-parameters remains a black art that requires years of experience to acquire. This report proposes several efficient ways to set the hyper-parameters that significantly reduce training time and improves performance. Specifically, this report shows how to examine the training validation/test loss function for subtle clues of underfitting and overfitting and suggests guidelines for moving toward the optimal balance point. Then it discusses how to increase/decrease the learning rate/momentum to speed up training. Our experiments show that it is crucial to balance every manner of regularization for each dataset and architecture. Weight decay is used as a sample regularizer to show how its optimal value is tightly coupled with the learning rates and momentums. Files to help replicate the results reported here are available.

Key facts. Cited by 223 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 52; 2022: 45; 2023: 47; 2024: 37; 2025: 31; 2026: 11.

223 papers.

Sort:

By area:

  1. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  2. 2021ImageNet-21K Pretraining for the Massescited by 607
  3. 2021Categorical Depth Distribution Network for Monocular 3D Object Detectioncited by 326
  4. 2021Query2Label: A Simple Transformer Way to Multi-Label Classificationcited by 126
  5. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  6. 2021M3DeTR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformerscited by 74
  7. 2022Point Density-Aware Voxels for LiDAR 3D Object Detectioncited by 70
  8. 2022πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimizationcited by 69
  9. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  10. 2022Transformers in Time-series Analysis: A Tutorialcited by 45
  11. 2024OmniJet-α: The first cross-task foundation model for particle physicscited by 41
  12. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
38 more
  1. 2023Learning Occupancy for Monocular 3D Object Detectioncited by 35
  2. 2021One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Visioncited by 33
  3. 2022Constrained Monotonic Neural Networkscited by 31
  4. 2023Revisiting Domain-Adaptive 3D Object Detection by Reliable, Diverse and Class-balanced Pseudo-Labelingcited by 28
  5. 2023Towards Robust Robot 3D Perception in Urban Environments: The UT Campus Object Datasetcited by 28
  6. 2022Homogeneous Multi-modal Feature Fusion and Interaction for 3D Object Detectioncited by 26
  7. 2024B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasonerscited by 26
  8. 2021Multi-label Classification with Partial Annotations using Class-aware Selective Losscited by 25
  9. 2021Data Augmentation in Natural Language Processing: A Novel Text Generation Approach for Long and Short Text Classifierscited by 24
  10. 2021Semantic Diversity Learning for Zero-Shot Multi-label Classificationcited by 24
  11. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 23
  12. 2022AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuningcited by 22
  13. 2023A General Framework for User-Guided Bayesian Optimizationcited by 22
  14. 2021MlTr: Multi-label Classification with Transformercited by 21
  15. 2022MAELi: Masked Autoencoder for Large-Scale LiDAR Point Cloudscited by 20
  16. 2023CrysMMNet: Multimodal Representation for Crystal Property Predictioncited by 18
  17. 2024CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scalecited by 18
  18. 2022Dense Voxel Fusion for 3D Object Detectioncited by 17
  19. 2022Bootstrap Generalization Ability from Loss Landscape Perspectivecited by 16
  20. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  21. 2021A Bayesian neural network predicts the dissolution of compact planetary systemscited by 15
  22. 2023Isolated Sign Language Recognition based on Tree Structure Skeleton Imagescited by 15
  23. 2021AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Flycited by 14
  24. 2023MDT3D: Multi-Dataset Training for LiDAR 3D Object Detection Generalizationcited by 14
  25. 2021SA-Det3D: Self-Attention Based Context-Aware 3D Object Detectioncited by 13
  26. 2021Symmetry meets AIcited by 13
  27. 2021Perceptual Loss for Robust Unsupervised Homography Estimationcited by 13
  28. 2021Adaptive Informative Path Planning Using Deep Reinforcement Learning for UAV-based Active Sensingcited by 12
  29. 2022AttentionHTR: Handwritten Text Recognition Based on Attention Encoder-Decoder Networkscited by 12
  30. 2023Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognitioncited by 12
  31. 2022NeuralTailor: Reconstructing Sewing Pattern Structures from 3D Point Clouds of Garmentscited by 10
  32. 2023CAMM: Building Category-Agnostic and Animatable 3D Models from Monocular Videoscited by 10
  33. 2021Data Augmented 3D Semantic Scene Completion with 2D Segmentation Priorscited by 9
  34. 2022Assessing Demographic Bias Transfer from Dataset to Model: A Case Study in Facial Expression Recognitioncited by 9
  35. 2022Adaptive Smoothness-weighted Adversarial Training for Multiple Perturbations with Its Stability Analysiscited by 9
  36. 2023Phase transitions in the mini-batch size for sparse and dense two-layer neural networkscited by 9
  37. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  38. 2021REX: Revisiting Budgeted Training with an Improved Schedulecited by 8

50 most cited of 223 All 223 (JSON)