# Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule

arXiv: https://arxiv.org/abs/2003.03977 · HTML: https://mlsanity.com/paper/2003.03977/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2020.

## Built on by (most cited first, 25 of 25)

- [No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models](https://mlsanity.com/paper/2307.06440/) (2023)
- [Ranger21: a synergistic deep learning optimizer](https://mlsanity.com/paper/2106.13731/) (2021)
- [Mid-Training of Large Language Models: A Survey](https://mlsanity.com/paper/2510.06826/) (2025)
- [Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes](https://mlsanity.com/paper/2209.03695/) (2022)
- [Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation](https://mlsanity.com/paper/2404.14827/) (2024)
- [Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler](https://mlsanity.com/paper/2508.01483/) (2025)
- [Who Are We Missing? A Principled Approach to Characterizing the Underrepresented Population](https://mlsanity.com/paper/2401.14512/) (2024)
- [Where Do Large Learning Rates Lead Us?](https://mlsanity.com/paper/2410.22113/) (2024)
- [Relaxed Attention for Transformer Models](https://mlsanity.com/paper/2209.09735/) (2022)
- [IMDeception: Grouped Information Distilling Super-Resolution Network](https://mlsanity.com/paper/2204.11463/) (2022)
- [A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Models](https://mlsanity.com/paper/2410.04103/) (2024)
- [Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules](https://mlsanity.com/paper/2509.18396/) (2025)
- [Efficient Multi-Purpose Cross-Attention Based Image Alignment Block for Edge Devices](https://mlsanity.com/paper/2206.00291/) (2022)
- [No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets](https://mlsanity.com/paper/2309.01694/) (2023)
- [Large Learning Rates Improve Generalization: But How Large Are We Talking About?](https://mlsanity.com/paper/2311.11303/) (2023)
- [Learning to Deliver: a Foundation Model for the Montreal Capacitated Vehicle Routing Problem](https://mlsanity.com/paper/2403.00026/) (2024)
- [Landscaper: Understanding Loss Landscapes Through Multi-Dimensional Topological Analysis](https://mlsanity.com/paper/2602.07135/) (2026)
- [LRTuner: A Learning Rate Tuner for Deep Neural Networks](https://mlsanity.com/paper/2105.14526/) (2021)
- [Distance Learner: Incorporating Manifold Prior to Model Training](https://mlsanity.com/paper/2207.06888/) (2022)
- [Unraveling Key Factors of Knowledge Distillation](https://mlsanity.com/paper/2312.08585/) (2023)
- [Randomness Helps Rigor: A Probabilistic Learning Rate Scheduler Bridging Theory and Deep Learning Practice](https://mlsanity.com/paper/2407.07613/) (2024)
- [Generalization of the Gibbs algorithm with high probability at low temperatures](https://mlsanity.com/paper/2502.11071/) (2025)
- [Escaping The Big Data Paradigm in Self-Supervised Representation Learning](https://mlsanity.com/paper/2502.18056/) (2025)
- [Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalization](https://mlsanity.com/paper/2604.04218/) (2026)
- [Neural Cluster First, Route Second: Capacitated Vehicle Routing via Differentiable Optimal Transport](https://mlsanity.com/paper/2605.09301/) (2026)

## Abstract (arXiv)

Several papers argue that wide minima generalize better than narrow minima. In this paper, through detailed experiments that not only corroborate the generalization properties of wide minima, we also provide empirical evidence for a new hypothesis that the density of wide minima is likely lower than the density of narrow minima. Further, motivated by this hypothesis, we design a novel explore-exploit learning rate schedule. On a variety of image and natural language datasets, compared to their original hand-tuned learning rate baselines, we show that our explore-exploit schedule can result in either up to 0.84% higher absolute accuracy using the original training budget or up to 57% reduced training time while achieving the original reported accuracy. For example, we achieve state-of-the-art (SOTA) accuracy for IWSLT'14 (DE-EN) dataset by just modifying the learning rate schedule of a high performing model.