{"total":117,"rows":[{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019,"area":null},{"id":"2304.01373","title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","created":"2023-04-03","cited_by_count":1786,"area":null},{"id":"2401.02954","title":"DeepSeek LLM: Scaling Open-Source Language Models with Longtermism","created":"2024-01-05","cited_by_count":642,"area":"lr"},{"id":"2409.11321","title":"SOAP: Improving and Stabilizing Shampoo using Adam","created":"2024-09-17","cited_by_count":192,"area":null},{"id":"2101.12176","title":"On the Origin of Implicit Regularization in Stochastic Gradient Descent","created":"2021-01-28","cited_by_count":184,"area":null},{"id":"2406.16793","title":"Adam-mini: Use Fewer Learning Rates To Gain More","created":"2024-06-24","cited_by_count":137,"area":null},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124,"area":null},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114,"area":null},{"id":"2406.19146","title":"Resolving Discrepancies in Compute-Optimal Scaling of Language Models","created":"2024-06-27","cited_by_count":88,"area":null},{"id":"2505.02222","title":"Practical Efficiency of Muon for Pretraining","created":"2025-05-04","cited_by_count":80,"area":null},{"id":"2410.21676","title":"How Does Critical Batch Size Scale in Pre-training?","created":"2024-10-29","cited_by_count":73,"area":null},{"id":"2109.08215","title":"Pre-trained Gaussian Processes for Bayesian Optimization","created":"2021-09-16","cited_by_count":63,"area":null},{"id":"2109.14119","title":"Stochastic Training is Not Necessary for Generalization","created":"2021-09-29","cited_by_count":63,"area":null},{"id":"2303.03108","title":"Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization","created":"2023-03-03","cited_by_count":57,"area":null},{"id":"2211.13672","title":"A Self-Attention Ansatz for Ab-initio Quantum Chemistry","created":"2022-11-24","cited_by_count":49,"area":null},{"id":"2311.00636","title":"Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures","created":"2023-11-01","cited_by_count":49,"area":null},{"id":"2405.13698","title":"How to set AdamW's weight decay as you scale model and dataset size","created":"2024-05-22","cited_by_count":43,"area":null},{"id":"2505.13738","title":"Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training","created":"2025-05-19","cited_by_count":43,"area":null},{"id":"2202.02831","title":"Anticorrelated Noise Injection for Improved Generalization","created":"2022-02-06","cited_by_count":42,"area":null},{"id":"2507.07101","title":"Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful","created":"2025-07-09","cited_by_count":41,"area":null},{"id":"2110.04369","title":"A Loss Curvature Perspective on Training Instability in Deep Learning","created":"2021-10-08","cited_by_count":40,"area":null},{"id":"2107.10211","title":"Differentiable Annealed Importance Sampling and the Perils of Gradient Noise","created":"2021-07-21","cited_by_count":36,"area":null},{"id":"2110.00641","title":"Batch size-invariance for policy optimization","created":"2021-10-01","cited_by_count":35,"area":null},{"id":"2105.13343","title":"Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error","created":"2021-05-27","cited_by_count":33,"area":null},{"id":"2410.13850","title":"Influence Functions for Scalable Data Attribution in Diffusion Models","created":"2024-10-17","cited_by_count":33,"area":"diffusion"},{"id":"2102.09468","title":"Near-optimal Local Convergence of Alternating Gradient Descent-Ascent for Minimax Optimization","created":"2021-02-18","cited_by_count":31,"area":null},{"id":"2209.11208","title":"A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases","created":"2022-09-22","cited_by_count":30,"area":null},{"id":"2103.17182","title":"Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization","created":"2021-03-31","cited_by_count":28,"area":null},{"id":"2104.11044","title":"Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes","created":"2021-04-22","cited_by_count":28,"area":null},{"id":"2203.08120","title":"Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers","created":"2022-03-15","cited_by_count":28,"area":null},{"id":"2507.01598","title":"Convergence Bound and Critical Batch Size of Muon Optimizer","created":"2025-07-02","cited_by_count":28,"area":null},{"id":"2106.00958","title":"A Generalizable Approach to Learning Optimizers","created":"2021-06-02","cited_by_count":23,"area":null},{"id":"2112.13246","title":"Towards Federated Learning on Time-Evolving Heterogeneous Data","created":"2021-12-25","cited_by_count":23,"area":null},{"id":"2505.23971","title":"Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training","created":"2025-05-29","cited_by_count":23,"area":null},{"id":"2410.23922","title":"Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training","created":"2024-10-31","cited_by_count":22,"area":null},{"id":"2507.02119","title":"Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks","created":"2025-07-02","cited_by_count":22,"area":null},{"id":"2512.05620","title":"Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales","created":"2025-12-05","cited_by_count":21,"area":null},{"id":"2102.06356","title":"A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes","created":"2021-02-12","cited_by_count":20,"area":null},{"id":"2107.09133","title":"The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion","created":"2021-07-19","cited_by_count":19,"area":null},{"id":"2206.01029","title":"Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions","created":"2022-06-02","cited_by_count":19,"area":null},{"id":"2505.23725","title":"MuLoCo: Muon is a practical inner optimizer for DiLoCo","created":"2025-05-29","cited_by_count":18,"area":null},{"id":"2205.11787","title":"Quadratic models for understanding catapult dynamics of neural networks","created":"2022-05-24","cited_by_count":17,"area":null},{"id":"2510.09378","title":"The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton","created":"2025-10-10","cited_by_count":17,"area":null},{"id":"2106.03696","title":"Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models","created":"2021-06-07","cited_by_count":16,"area":null},{"id":"2201.13011","title":"On the Power-Law Hessian Spectrums in Deep Learning","created":"2022-01-31","cited_by_count":16,"area":null},{"id":"2110.14109","title":"Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums","created":"2021-10-27","cited_by_count":15,"area":null},{"id":"2607.21716","title":"A Defense of the Quadratic Model","created":"2026-07-23","cited_by_count":14,"area":null},{"id":"2103.01338","title":"Acceleration via Fractal Learning Rate Schedules","created":"2021-03-01","cited_by_count":13,"area":null},{"id":"2305.13209","title":"Faster Differentially Private Convex Optimization via Second-Order Methods","created":"2023-05-22","cited_by_count":13,"area":null},{"id":"2404.09816","title":"FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity","created":"2024-04-15","cited_by_count":13,"area":null},{"id":"2505.02809","title":"Towards Quantifying the Hessian Structure of Neural Networks","created":"2025-05-05","cited_by_count":13,"area":null},{"id":"2409.08770","title":"Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent","created":"2024-09-13","cited_by_count":11,"area":null},{"id":"2111.00856","title":"Large-Scale Deep Learning Optimizations: A Comprehensive Survey","created":"2021-11-01","cited_by_count":10,"area":null},{"id":"2309.11197","title":"The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute","created":"2023-09-20","cited_by_count":10,"area":null},{"id":"2111.05968","title":"Linear Speedup in Personalized Collaborative Learning","created":"2021-11-10","cited_by_count":9,"area":null},{"id":"2203.00089","title":"Amortized Proximal Optimization","created":"2022-02-28","cited_by_count":9,"area":null},{"id":"2301.13703","title":"Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning","created":"2023-01-31","cited_by_count":9,"area":null},{"id":"2312.05705","title":"Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC","created":"2023-12-09","cited_by_count":9,"area":null},{"id":"2410.05838","title":"Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit","created":"2024-10-08","cited_by_count":9,"area":null},{"id":"2505.12387","title":"Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning","created":"2025-05-18","cited_by_count":9,"area":null},{"id":"2509.25087","title":"Scaling with Collapse: Efficient and Predictable Training of LLM Families","created":"2025-09-29","cited_by_count":9,"area":null},{"id":"2110.09057","title":"Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimization","created":"2021-10-18","cited_by_count":8,"area":null},{"id":"2201.11989","title":"Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule","created":"2022-01-28","cited_by_count":8,"area":null},{"id":"2402.11215","title":"AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods","created":"2024-02-17","cited_by_count":8,"area":null},{"id":"2402.18752","title":"Pre-training Differentially Private Models with Limited Public Data","created":"2024-02-28","cited_by_count":8,"area":null},{"id":"2211.13403","title":"Differentially Private Image Classification from Features","created":"2022-11-24","cited_by_count":7,"area":null},{"id":"2411.00999","title":"Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers","created":"2024-11-01","cited_by_count":7,"area":null},{"id":"2605.19095","title":"ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models","created":"2026-05-18","cited_by_count":7,"area":null},{"id":"2306.16484","title":"Towards a Better Theoretical Understanding of Independent Subnetwork Training","created":"2023-06-28","cited_by_count":6,"area":null},{"id":"2510.14717","title":"Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling","created":"2025-10-16","cited_by_count":6,"area":null},{"id":"2511.21622","title":"On the Origin of Algorithmic Progress in AI","created":"2025-11-26","cited_by_count":6,"area":null},{"id":"2209.09162","title":"On the Theoretical Properties of Noise Correlation in Stochastic Optimization","created":"2022-09-19","cited_by_count":5,"area":null},{"id":"2402.01514","title":"Mapping the Multiverse of Latent Representations","created":"2024-02-02","cited_by_count":5,"area":null},{"id":"2402.18824","title":"Batch size invariant Adam","created":"2024-02-29","cited_by_count":5,"area":null},{"id":"2409.19946","title":"Illustrious: an Open Advanced Illustration Model","created":"2024-09-30","cited_by_count":5,"area":null},{"id":"2109.09833","title":"Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics","created":"2021-09-20","cited_by_count":4,"area":null},{"id":"2206.13290","title":"Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness","created":"2022-06-27","cited_by_count":4,"area":null},{"id":"2306.00342","title":"Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks","created":"2023-06-01","cited_by_count":4,"area":null},{"id":"2211.08583","title":"Empirical Study on Optimizer Selection for Out-of-Distribution Generalization","created":"2022-11-15","cited_by_count":3,"area":null},{"id":"2311.17539","title":"Critical Influence of Overparameterization on Sharpness-aware Minimization","created":"2023-11-29","cited_by_count":3,"area":null},{"id":"2402.05264","title":"AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size","created":"2024-02-07","cited_by_count":3,"area":null},{"id":"2508.00180","title":"EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes","created":"2025-07-31","cited_by_count":3,"area":null},{"id":"2602.20433","title":"Disentangling Geometry, Performance, and Training in Language Models","created":"2026-02-24","cited_by_count":3,"area":null},{"id":"2202.06232","title":"Generalized Tangent Kernel: A Unified Geometric Foundation for Natural Gradient and Standard Gradient","created":"2022-02-13","cited_by_count":2,"area":null},{"id":"2306.05300","title":"Anti-Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances in Flat Directions","created":"2023-06-08","cited_by_count":2,"area":null},{"id":"2310.14963","title":"Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lens","created":"2023-10-23","cited_by_count":2,"area":null},{"id":"2501.08883","title":"Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentum","created":"2025-01-15","cited_by_count":2,"area":null},{"id":"2501.18164","title":"Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size","created":"2025-01-30","cited_by_count":2,"area":null},{"id":"2506.12903","title":"Variational Learning Finds Flatter Solutions at the Edge of Stability","created":"2025-06-15","cited_by_count":2,"area":null},{"id":"2509.25380","title":"Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs","created":"2025-09-29","cited_by_count":2,"area":null},{"id":"2510.04686","title":"How does the optimizer implicitly bias the model merging loss landscape?","created":"2025-10-06","cited_by_count":2,"area":null},{"id":"2606.19179","title":"Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods","created":"2026-06-17","cited_by_count":2,"area":null},{"id":"2106.02713","title":"Learning Curves for SGD on Structured Features","created":"2021-06-04","cited_by_count":1,"area":null},{"id":"2108.11713","title":"The Number of Steps Needed for Nonconvex Optimization of a Deep Learning Optimizer is a Rational Function of Batch Size","created":"2021-08-26","cited_by_count":1,"area":null},{"id":"2208.09814","title":"Critical Bach Size Minimizes Stochastic First-Order Oracle Complexity of Deep Learning Optimizer using Hyperparameters Close to One","created":"2022-08-21","cited_by_count":1,"area":null},{"id":"2305.01377","title":"Random Function Descent","created":"2023-05-02","cited_by_count":1,"area":null},{"id":"2402.13810","title":"The Expected Loss of Preconditioned Langevin Dynamics Reveals the Hessian Rank","created":"2024-02-21","cited_by_count":1,"area":null},{"id":"2502.18323","title":"Accelerated Training on Low-Power Edge Devices","created":"2025-02-25","cited_by_count":1,"area":null},{"id":"2505.18107","title":"Accelerating Learned Image Compression Through Modeling Neural Training Dynamics","created":"2025-05-23","cited_by_count":1,"area":null},{"id":"2606.14259","title":"Beyond a Single Explanation of the Adam--SGD Gap","created":"2026-06-12","cited_by_count":1,"area":null}],"next_cursor":"100"}