{"total":56,"rows":[{"id":"2103.00065","title":"Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability","created":"2021-02-26","cited_by_count":427,"area":null},{"id":"2110.06914","title":"What Happens after SGD Reaches Zero Loss? --A Mathematical Framework","created":"2021-10-13","cited_by_count":107,"area":null},{"id":"2205.10287","title":"On the SDEs and Scaling Rules for Adaptive Gradient Algorithms","created":"2022-05-20","cited_by_count":104,"area":null},{"id":"2302.07011","title":"A Modern Look at the Relationship between Sharpness and Generalization","created":"2023-02-14","cited_by_count":92,"area":null},{"id":"2206.07085","title":"Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction","created":"2022-06-14","cited_by_count":86,"area":null},{"id":"2102.12470","title":"On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)","created":"2021-02-24","cited_by_count":73,"area":null},{"id":"2103.14636","title":"A Practical Survey on Faster and Lighter Transformers","created":"2021-03-26","cited_by_count":73,"area":null},{"id":"2310.04415","title":"Why Do We Need Weight Decay in Modern Deep Learning?","created":"2023-10-06","cited_by_count":72,"area":null},{"id":"2210.05337","title":"SGD with Large Step Sizes Learns Sparse Features","created":"2022-10-11","cited_by_count":68,"area":null},{"id":"2109.14119","title":"Stochastic Training is Not Necessary for Generalization","created":"2021-09-29","cited_by_count":63,"area":null},{"id":"2404.04454","title":"Implicit Bias of AdamW: $\\ell_\\infty$ Norm Constrained Optimization","created":"2024-04-05","cited_by_count":58,"area":null},{"id":"2305.17212","title":"Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks","created":"2023-05-26","cited_by_count":53,"area":null},{"id":"2105.02716","title":"Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks","created":"2021-05-06","cited_by_count":50,"area":null},{"id":"2405.13698","title":"How to set AdamW's weight decay as you scale model and dataset size","created":"2024-05-22","cited_by_count":43,"area":null},{"id":"2502.15938","title":"Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs","created":"2025-02-21","cited_by_count":35,"area":null},{"id":"2202.00980","title":"Robust Training of Neural Networks Using Scale Invariant Architectures","created":"2022-02-02","cited_by_count":29,"area":null},{"id":"2206.08900","title":"Adapting the Linearised Laplace Model Evidence for Modern Deep Learning","created":"2022-06-17","cited_by_count":29,"area":null},{"id":"2301.08203","title":"An SDE for Modeling SAM: Theory and Insights","created":"2023-01-19","cited_by_count":22,"area":null},{"id":"2410.23922","title":"Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training","created":"2024-10-31","cited_by_count":22,"area":null},{"id":"2106.15739","title":"On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay","created":"2021-06-29","cited_by_count":20,"area":null},{"id":"2510.19093","title":"Weight Decay may matter more than muP for Learning Rate Transfer in Practice","created":"2025-10-21","cited_by_count":20,"area":null},{"id":"2107.09133","title":"The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion","created":"2021-07-19","cited_by_count":19,"area":null},{"id":"2209.03695","title":"Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes","created":"2022-09-08","cited_by_count":19,"area":null},{"id":"2402.07193","title":"Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent","created":"2024-02-11","cited_by_count":19,"area":null},{"id":"2303.11135","title":"TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization","created":"2023-03-20","cited_by_count":18,"area":null},{"id":"2403.04081","title":"Directional Smoothness and Gradient Methods: Convergence and Adaptivity","created":"2024-03-06","cited_by_count":17,"area":null},{"id":"2606.16899","title":"Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization","created":"2026-06-15","cited_by_count":16,"area":null},{"id":"2110.06256","title":"Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspective","created":"2021-10-12","cited_by_count":15,"area":null},{"id":"2310.11428","title":"Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression","created":"2023-10-17","cited_by_count":14,"area":null},{"id":"2502.02496","title":"Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries","created":"2025-02-04","cited_by_count":14,"area":null},{"id":"2103.01338","title":"Acceleration via Fractal Learning Rate Schedules","created":"2021-03-01","cited_by_count":13,"area":null},{"id":"2210.15898","title":"Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis","created":"2022-10-28","cited_by_count":13,"area":null},{"id":"2502.05300","title":"Parameter Symmetry Potentially Unifies Deep Learning Theory","created":"2025-02-07","cited_by_count":12,"area":null},{"id":"2103.12682","title":"How to decay your learning rate","created":"2021-03-23","cited_by_count":11,"area":null},{"id":"2212.07295","title":"Maximal Initial Learning Rates in Deep ReLU Networks","created":"2022-12-14","cited_by_count":11,"area":null},{"id":"2309.04644","title":"Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decay","created":"2023-09-09","cited_by_count":9,"area":null},{"id":"2401.11600","title":"Understanding the Generalization Benefits of Late Learning Rate Decay","created":"2024-01-21","cited_by_count":6,"area":null},{"id":"2410.22113","title":"Where Do Large Learning Rates Lead Us?","created":"2024-10-29","cited_by_count":6,"area":null},{"id":"2305.15912","title":"Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks","created":"2023-05-25","cited_by_count":5,"area":null},{"id":"2311.14387","title":"Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescaling","created":"2023-11-24","cited_by_count":4,"area":null},{"id":"2410.11985","title":"The Fair Language Model Paradox","created":"2024-10-15","cited_by_count":4,"area":null},{"id":"2602.11137","title":"Weight Decay Improves Language Model Plasticity","created":"2026-02-11","cited_by_count":4,"area":null},{"id":"2607.22444","title":"Hyperball May Not Be a Free Lunch","created":"2026-07-24","cited_by_count":4,"area":null},{"id":"2302.12444","title":"On the Training Instability of Shuffling SGD with Batch Normalization","created":"2023-02-24","cited_by_count":3,"area":null},{"id":"2608.24814","title":"Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining","created":"2026-08-25","cited_by_count":3,"area":null},{"id":"2606.29158","title":"On the Nonlinearity of Learning Rate Scaling for LLM Training","created":"2026-06-28","cited_by_count":2,"area":null},{"id":"2309.01694","title":"No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets","created":"2023-09-04","cited_by_count":1,"area":null},{"id":"2311.11303","title":"Large Learning Rates Improve Generalization: But How Large Are We Talking About?","created":"2023-11-19","cited_by_count":1,"area":null},{"id":"2605.04418","title":"Demystifying Manifold Constraints in LLM Pre-training","created":"2026-05-06","cited_by_count":1,"area":null},{"id":"2607.21005","title":"Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay","created":"2026-07-23","cited_by_count":1,"area":null},{"id":"2511.07308","title":"Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?","created":"2025-11-10","cited_by_count":0,"area":null},{"id":"2605.09154","title":"Predicting Large Model Test Losses with a Noisy Quadratic System","created":"2026-05-09","cited_by_count":0,"area":null},{"id":"2605.29152","title":"Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias","created":"2026-05-27","cited_by_count":0,"area":null},{"id":"2609.07017","title":"HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball","created":"2026-09-07","cited_by_count":0,"area":null},{"id":"2609.09116","title":"When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay","created":"2026-09-08","cited_by_count":0,"area":null},{"id":"2609.32290","title":"A Journey to the Edge of Stability","created":"2026-09-26","cited_by_count":0,"area":null}],"next_cursor":null}