{"list":"other","total":77,"rows":[{"id":"1412.6980","title":"Adam: A Method for Stochastic Optimization","created":"2014-12-22","cited_by_count":37240},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"2307.09288","title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","created":"2023-07-18","cited_by_count":13028},{"id":"2205.01068","title":"OPT: Open Pre-trained Transformer Language Models","created":"2022-05-02","cited_by_count":3512},{"id":"1609.07843","title":"Pointer Sentinel Mixture Models","created":"2016-09-26","cited_by_count":3096},{"id":"1901.02860","title":"Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context","created":"2019-01-09","cited_by_count":1850},{"id":"2304.01373","title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","created":"2023-04-03","cited_by_count":1786},{"id":"2010.01412","title":"Sharpness-Aware Minimization for Efficiently Improving Generalization","created":"2020-10-03","cited_by_count":1417},{"id":"1609.04836","title":"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima","created":"2016-09-15","cited_by_count":1352},{"id":"2305.10403","title":"PaLM 2 Technical Report","created":"2023-05-17","cited_by_count":1162},{"id":"1804.04235","title":"Adafactor: Adaptive Learning Rates with Sublinear Memory Cost","created":"2018-04-11","cited_by_count":905},{"id":"2210.02414","title":"GLM-130B: An Open Bilingual Pre-trained Model","created":"2022-10-05","cited_by_count":843},{"id":"1710.10345","title":"The Implicit Bias of Gradient Descent on Separable Data","created":"2017-10-27","cited_by_count":655},{"id":"1912.02178","title":"Fantastic Generalization Measures and Where to Find Them","created":"2019-12-04","cited_by_count":513},{"id":"1703.04933","title":"Sharp Minima Can Generalize For Deep Nets","created":"2017-03-15","cited_by_count":466},{"id":"2406.11794","title":"DataComp-LM: In search of the next generation of training sets for language models","created":"2024-06-17","cited_by_count":398},{"id":"1906.05890","title":"Gradient Descent Maximizes the Margin of Homogeneous Neural Networks","created":"2019-06-13","cited_by_count":320},{"id":"2102.11600","title":"ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks","created":"2021-02-23","cited_by_count":293},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"1802.08246","title":"Characterizing Implicit Bias in Terms of Optimization Geometry","created":"2018-02-22","cited_by_count":266},{"id":"1806.00468","title":"Implicit Bias of Gradient Descent on Linear Convolutional Networks","created":"2018-06-01","cited_by_count":251},{"id":"2101.12176","title":"On the Origin of Implicit Regularization in Stochastic Gradient Descent","created":"2021-01-28","cited_by_count":184},{"id":"1810.02032","title":"Gradient descent aligns the layers of deep linear networks","created":"2018-10-04","cited_by_count":182},{"id":"2006.06657","title":"Directional convergence and alignment in deep learning","created":"2020-06-11","cited_by_count":178},{"id":"2009.11162","title":"Implicit Gradient Regularization","created":"2020-09-23","cited_by_count":146},{"id":"2203.02714","title":"Towards Efficient and Scalable Sharpness-Aware Minimization","created":"2022-03-05","cited_by_count":129},{"id":"2205.09745","title":"Understanding Gradient Descent on Edge of Stability in Deep Learning","created":"2022-05-19","cited_by_count":128},{"id":"2010.05627","title":"Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning","created":"2020-10-12","cited_by_count":126},{"id":"2106.06530","title":"Label Noise SGD Provably Prefers Flat Global Minimizers","created":"2021-06-11","cited_by_count":121},{"id":"2007.01547","title":"Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers","created":"2020-07-03","cited_by_count":119},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114},{"id":"1810.12281","title":"Three Mechanisms of Weight Decay Regularization","created":"2018-10-29","cited_by_count":113},{"id":"1803.01905","title":"Convergence of Gradient Descent on Separable Data","created":"2018-03-05","cited_by_count":110},{"id":"2205.10287","title":"On the SDEs and Scaling Rules for Adaptive Gradient Algorithms","created":"2022-05-20","cited_by_count":104},{"id":"2509.02046","title":"Fantastic Pretraining Optimizers and Where to Find Them","created":"2025-09-02","cited_by_count":100},{"id":"2205.14083","title":"Sharpness-Aware Training for Free","created":"2022-05-27","cited_by_count":95},{"id":"2302.07011","title":"A Modern Look at the Relationship between Sharpness and Generalization","created":"2023-02-14","cited_by_count":92},{"id":"2406.19146","title":"Resolving Discrepancies in Compute-Optimal Scaling of Language Models","created":"2024-06-27","cited_by_count":88},{"id":"1803.07300","title":"Risk and parameter convergence of logistic regression","created":"2018-03-20","cited_by_count":85},{"id":"2206.04920","title":"Fisher SAM: Information Geometry and Sharpness Aware Minimisation","created":"2022-06-10","cited_by_count":84},{"id":"2010.04925","title":"Regularizing Neural Networks via Adversarial Model Perturbation","created":"2020-10-10","cited_by_count":78},{"id":"2410.21676","title":"How Does Critical Batch Size Scale in Pre-training?","created":"2024-10-29","cited_by_count":73},{"id":"2310.04415","title":"Why Do We Need Weight Decay in Modern Deep Learning?","created":"2023-10-06","cited_by_count":72},{"id":"1806.01796","title":"Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate","created":"2018-06-05","cited_by_count":64},{"id":"1901.04653","title":"Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis","created":"2019-01-15","cited_by_count":59},{"id":"2404.04454","title":"Implicit Bias of AdamW: $\\ell_\\infty$ Norm Constrained Optimization","created":"2024-04-05","cited_by_count":58},{"id":"2503.19206","title":"Overtrained Language Models Are Harder to Fine-Tune","created":"2025-03-24","cited_by_count":58},{"id":"2409.03137","title":"The AdEMAMix Optimizer: Better, Faster, Older","created":"2024-09-05","cited_by_count":54},{"id":"1905.07325","title":"Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models","created":"2019-05-17","cited_by_count":53},{"id":"2403.12350","title":"Friendly Sharpness-Aware Minimization","created":"2024-03-19","cited_by_count":53},{"id":"2012.06244","title":"The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks","created":"2020-12-11","cited_by_count":46},{"id":"2202.00089","title":"Understanding AdamW through Proximal Methods and Scale-Freeness","created":"2022-01-31","cited_by_count":46},{"id":"2507.07101","title":"Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful","created":"2025-07-09","cited_by_count":41},{"id":"2006.15815","title":"Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum","created":"2020-06-29","cited_by_count":40},{"id":"2505.21829","title":"In Search of Adam's Secret Sauce","created":"2025-05-27","cited_by_count":40},{"id":"2309.15639","title":"Enhancing Sharpness-Aware Optimization Through Variance Suppression","created":"2023-09-27","cited_by_count":37},{"id":"2009.06125","title":"A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms","created":"2020-09-14","cited_by_count":33},{"id":"1906.04285","title":"Continuous Time Analysis of Momentum Methods","created":"2019-06-10","cited_by_count":32},{"id":"2509.14786","title":"Pre-training under infinite compute","created":"2025-09-18","cited_by_count":29},{"id":"2309.00079","title":"On the Implicit Bias of Adam","created":"2023-08-31","cited_by_count":27},{"id":"2410.23819","title":"Weight decay induces low-rank attention layers","created":"2024-10-31","cited_by_count":27},{"id":"2302.00849","title":"Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent","created":"2023-02-02","cited_by_count":25},{"id":"2409.15156","title":"Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling","created":"2024-09-23","cited_by_count":23},{"id":"2411.15958","title":"Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise","created":"2024-11-24","cited_by_count":23},{"id":"2110.03891","title":"Does Momentum Change the Implicit Regularization on Separable Data?","created":"2021-10-08","cited_by_count":22},{"id":"1906.03559","title":"The Implicit Bias of AdaGrad on Separable Data","created":"2019-06-09","cited_by_count":16},{"id":"2406.03682","title":"A Universal Class of Sharpness-Aware Minimization Algorithms","created":"2024-06-06","cited_by_count":14},{"id":"2210.15898","title":"Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis","created":"2022-10-28","cited_by_count":13},{"id":"1903.02237","title":"Positively Scale-Invariant Flatness of ReLU Neural Networks","created":"2019-03-06","cited_by_count":11},{"id":"2302.01952","title":"On a continuous time model of gradient descent dynamics and instability in deep learning","created":"2023-02-03","cited_by_count":11},{"id":"2312.16143","title":"On the Trajectories of SGD Without Replacement","created":"2023-12-26","cited_by_count":11},{"id":"2410.10683","title":"SAMPa: Sharpness-aware Minimization Parallelized","created":"2024-10-14","cited_by_count":8},{"id":"2304.01335","title":"Charting the Topography of the Neural Network Landscape with Thermal-Like Noise","created":"2023-04-03","cited_by_count":6},{"id":"2502.02132","title":"How Memory in Optimization Algorithms Implicitly Modifies the Loss","created":"2025-02-04","cited_by_count":6},{"id":"1807.11354","title":"Multiscale analysis of accelerated gradient methods","created":"2018-07-30","cited_by_count":3},{"id":"2509.08483","title":"Modified Loss of Momentum Gradient Descent: Fine-Grained Analysis","created":"2025-09-10","cited_by_count":2}],"next_cursor":null}