{"total":193,"rows":[{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019,"area":null},{"id":"2401.02954","title":"DeepSeek LLM: Scaling Open-Source Language Models with Longtermism","created":"2024-01-05","cited_by_count":642,"area":"lr"},{"id":"2106.06843","title":"Federated Learning on Non-IID Data: A Survey","created":"2021-06-12","cited_by_count":346,"area":null},{"id":"2101.06840","title":"ZeRO-Offload: Democratizing Billion-Scale Model Training","created":"2021-01-18","cited_by_count":341,"area":null},{"id":"2203.03466","title":"Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer","created":"2022-03-07","cited_by_count":288,"area":null},{"id":"2204.13650","title":"Unlocking High-Accuracy Differentially Private Image Classification through Scale","created":"2022-04-28","cited_by_count":224,"area":null},{"id":"2106.06639","title":"Federated Learning with Buffered Asynchronous Aggregation","created":"2021-06-11","cited_by_count":190,"area":null},{"id":"2312.03863","title":"Efficient Large Language Models: A Survey","created":"2023-12-06","cited_by_count":186,"area":null},{"id":"2101.12176","title":"On the Origin of Implicit Regularization in Stochastic Gradient Descent","created":"2021-01-28","cited_by_count":184,"area":null},{"id":"2203.02714","title":"Towards Efficient and Scalable Sharpness-Aware Minimization","created":"2022-03-05","cited_by_count":129,"area":null},{"id":"2106.06530","title":"Label Noise SGD Provably Prefers Flat Global Minimizers","created":"2021-06-11","cited_by_count":121,"area":null},{"id":"2108.01624","title":"Large-Scale Differentially Private BERT","created":"2021-08-03","cited_by_count":118,"area":null},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114,"area":null},{"id":"2304.03208","title":"Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster","created":"2023-04-06","cited_by_count":109,"area":null},{"id":"2406.19146","title":"Resolving Discrepancies in Compute-Optimal Scaling of Language Models","created":"2024-06-27","cited_by_count":88,"area":null},{"id":"2203.12533","title":"Pathways: Asynchronous Distributed Dataflow for ML","created":"2022-03-23","cited_by_count":82,"area":null},{"id":"2505.02222","title":"Practical Efficiency of Muon for Pretraining","created":"2025-05-04","cited_by_count":80,"area":null},{"id":"2106.07820","title":"On Large-Cohort Training for Federated Learning","created":"2021-06-15","cited_by_count":74,"area":null},{"id":"2102.12470","title":"On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)","created":"2021-02-24","cited_by_count":73,"area":null},{"id":"2410.21676","title":"How Does Critical Batch Size Scale in Pre-training?","created":"2024-10-29","cited_by_count":73,"area":null},{"id":"2109.14119","title":"Stochastic Training is Not Necessary for Generalization","created":"2021-09-29","cited_by_count":63,"area":null},{"id":"2305.10633","title":"Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Models","created":"2023-05-18","cited_by_count":61,"area":null},{"id":"2311.17295","title":"Elo Uncovered: Robustness and Best Practices in Language Model Evaluation","created":"2023-11-29","cited_by_count":61,"area":null},{"id":"2303.14177","title":"Scaling Expert Language Models with Unsupervised Domain Discovery","created":"2023-03-24","cited_by_count":60,"area":null},{"id":"2106.11872","title":"Randomness In Neural Network Training: Characterizing The Impact of Tooling","created":"2021-06-22","cited_by_count":59,"area":null},{"id":"2211.09760","title":"VeLO: Training Versatile Learned Optimizers by Scaling Up","created":"2022-11-17","cited_by_count":57,"area":null},{"id":"2305.17212","title":"Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks","created":"2023-05-26","cited_by_count":53,"area":null},{"id":"2102.04828","title":"Consensus Control for Decentralized Deep Learning","created":"2021-02-09","cited_by_count":49,"area":null},{"id":"2405.13698","title":"How to set AdamW's weight decay as you scale model and dataset size","created":"2024-05-22","cited_by_count":43,"area":null},{"id":"2505.13738","title":"Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training","created":"2025-05-19","cited_by_count":43,"area":null},{"id":"2407.07972","title":"Deconstructing What Makes a Good Optimizer for Language Models","created":"2024-07-10","cited_by_count":41,"area":null},{"id":"2507.07101","title":"Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful","created":"2025-07-09","cited_by_count":41,"area":null},{"id":"2110.04369","title":"A Loss Curvature Perspective on Training Instability in Deep Learning","created":"2021-10-08","cited_by_count":40,"area":null},{"id":"2307.02047","title":"CAME: Confidence-guided Adaptive Memory Efficient Optimization","created":"2023-07-05","cited_by_count":36,"area":null},{"id":"2310.03882","title":"Small batch deep reinforcement learning","created":"2023-10-05","cited_by_count":36,"area":null},{"id":"2110.00641","title":"Batch size-invariance for policy optimization","created":"2021-10-01","cited_by_count":35,"area":null},{"id":"2502.15938","title":"Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs","created":"2025-02-21","cited_by_count":35,"area":null},{"id":"2209.05310","title":"On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models","created":"2022-09-12","cited_by_count":34,"area":null},{"id":"2310.19726","title":"A Path to Simpler Models Starts With Noise","created":"2023-10-30","cited_by_count":33,"area":null},{"id":"2106.13897","title":"Implicit Gradient Alignment in Distributed and Federated Learning","created":"2021-06-25","cited_by_count":29,"area":null},{"id":"2604.21691","title":"There Will Be a Scientific Theory of Deep Learning","created":"2026-04-23","cited_by_count":29,"area":null},{"id":"2507.01598","title":"Convergence Bound and Critical Batch Size of Muon Optimizer","created":"2025-07-02","cited_by_count":28,"area":null},{"id":"2308.00852","title":"CASSINI: Network-Aware Job Scheduling in Machine Learning Clusters","created":"2023-08-01","cited_by_count":26,"area":null},{"id":"2203.10036","title":"On the Generalization Mystery in Deep Learning","created":"2022-03-18","cited_by_count":25,"area":null},{"id":"2403.06563","title":"Unraveling the Mystery of Scaling Laws: Part I","created":"2024-03-11","cited_by_count":24,"area":null},{"id":"2206.07252","title":"Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions","created":"2022-06-15","cited_by_count":23,"area":null},{"id":"2405.14578","title":"Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling","created":"2024-05-23","cited_by_count":22,"area":null},{"id":"2410.23922","title":"Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training","created":"2024-10-31","cited_by_count":22,"area":null},{"id":"2202.04598","title":"Reproducibility in Optimization: Theoretical Framework and Limits","created":"2022-02-09","cited_by_count":21,"area":null},{"id":"2102.06356","title":"A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes","created":"2021-02-12","cited_by_count":20,"area":null},{"id":"2110.08133","title":"Trade-offs of Local SGD at Scale: An Empirical Study","created":"2021-10-15","cited_by_count":20,"area":null},{"id":"2411.17470","title":"Towards Precise Scaling Laws for Video Diffusion Transformers","created":"2024-11-25","cited_by_count":20,"area":"diffusion"},{"id":"2306.09000","title":"When and Why Momentum Accelerates SGD:An Empirical Study","created":"2023-06-15","cited_by_count":19,"area":null},{"id":"2603.07685","title":"Scalable Training of Mixture-of-Experts Models with Megatron Core","created":"2026-03-08","cited_by_count":19,"area":"moe"},{"id":"2307.15196","title":"The Marginal Value of Momentum for Small Learning Rate SGD","created":"2023-07-27","cited_by_count":18,"area":null},{"id":"2404.19261","title":"High dimensional analysis reveals conservative sharpening and a stochastic edge of stability","created":"2024-04-30","cited_by_count":18,"area":null},{"id":"2505.23725","title":"MuLoCo: Muon is a practical inner optimizer for DiLoCo","created":"2025-05-29","cited_by_count":18,"area":null},{"id":"2103.02351","title":"Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates","created":"2021-03-03","cited_by_count":17,"area":null},{"id":"2402.16828","title":"Training Neural Networks from Scratch with Parallel Low-Rank Adapters","created":"2024-02-26","cited_by_count":17,"area":null},{"id":"2510.09378","title":"The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton","created":"2025-10-10","cited_by_count":17,"area":null},{"id":"2306.02913","title":"Decentralized SGD and Average-direction SAM are Asymptotically Equivalent","created":"2023-06-05","cited_by_count":15,"area":null},{"id":"2206.05891","title":"Anchor Sampling for Federated Learning with Partial Client Participation","created":"2022-06-13","cited_by_count":14,"area":null},{"id":"2103.01338","title":"Acceleration via Fractal Learning Rate Schedules","created":"2021-03-01","cited_by_count":13,"area":null},{"id":"2411.07854","title":"Tucano: Advancing Neural Text Generation for Portuguese","created":"2024-11-12","cited_by_count":13,"area":null},{"id":"2506.03404","title":"The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks","created":"2025-06-03","cited_by_count":13,"area":"deeprl"},{"id":"2203.10736","title":"The activity-weight duality in feed forward neural networks: The geometric determinants of generalization","created":"2022-03-21","cited_by_count":12,"area":null},{"id":"2305.12213","title":"Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching","created":"2023-05-20","cited_by_count":12,"area":null},{"id":"2309.10688","title":"On the different regimes of Stochastic Gradient Descent","created":"2023-09-19","cited_by_count":12,"area":null},{"id":"2103.12682","title":"How to decay your learning rate","created":"2021-03-23","cited_by_count":11,"area":null},{"id":"2106.00116","title":"Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images","created":"2021-05-31","cited_by_count":11,"area":null},{"id":"2106.00221","title":"Concurrent Adversarial Learning for Large-Batch Training","created":"2021-06-01","cited_by_count":11,"area":null},{"id":"2107.07572","title":"Globally Convergent Multilevel Training of Deep Residual Networks","created":"2021-07-15","cited_by_count":11,"area":null},{"id":"2206.05985","title":"Modeling the Machine Learning Multiverse","created":"2022-06-13","cited_by_count":11,"area":null},{"id":"2206.11124","title":"A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta","created":"2022-06-22","cited_by_count":11,"area":null},{"id":"2303.00586","title":"FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensembling","created":"2023-03-01","cited_by_count":11,"area":null},{"id":"2409.08770","title":"Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent","created":"2024-09-13","cited_by_count":11,"area":null},{"id":"2108.11976","title":"JUWELS Booster -- A Supercomputer for Large-Scale AI Research","created":"2021-06-30","cited_by_count":10,"area":null},{"id":"2111.00856","title":"Large-Scale Deep Learning Optimizations: A Comprehensive Survey","created":"2021-11-01","cited_by_count":10,"area":null},{"id":"2309.11197","title":"The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute","created":"2023-09-20","cited_by_count":10,"area":null},{"id":"2404.06114","title":"Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey","created":"2024-04-09","cited_by_count":10,"area":null},{"id":"2603.15958","title":"Deriving Hyperparameter Scaling Laws via Modern Optimization Theory","created":"2026-03-16","cited_by_count":10,"area":null},{"id":"2301.13703","title":"Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning","created":"2023-01-31","cited_by_count":9,"area":null},{"id":"2410.05838","title":"Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit","created":"2024-10-08","cited_by_count":9,"area":null},{"id":"2606.04101","title":"UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing","created":"2026-06-02","cited_by_count":9,"area":"moe"},{"id":"2111.09562","title":"COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression","created":"2021-11-18","cited_by_count":8,"area":null},{"id":"2201.11989","title":"Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule","created":"2022-01-28","cited_by_count":8,"area":null},{"id":"2202.04124","title":"A Mini-Block Fisher Method for Deep Neural Networks","created":"2022-02-08","cited_by_count":8,"area":null},{"id":"2304.02658","title":"Predictive Coding as a Neuromorphic Alternative to Backpropagation: A Critical Evaluation","created":"2023-04-05","cited_by_count":8,"area":null},{"id":"2402.11215","title":"AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods","created":"2024-02-17","cited_by_count":8,"area":null},{"id":"2503.05931","title":"Training and Inference Efficiency of Encoder-Decoder Speech Models","created":"2025-03-07","cited_by_count":8,"area":null},{"id":"2512.13472","title":"Scaling Laws for Code: Every Programming Language Matters","created":"2025-12-15","cited_by_count":8,"area":null},{"id":"2102.02344","title":"Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models","created":"2021-02-03","cited_by_count":7,"area":null},{"id":"2206.15143","title":"Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning","created":"2022-06-30","cited_by_count":7,"area":null},{"id":"2210.14312","title":"JAX-DIPS: Neural bootstrapping of finite discretization methods and application to elliptic problems with discontinuities","created":"2022-10-25","cited_by_count":7,"area":null},{"id":"2311.17094","title":"In Search of a Data Transformation That Accelerates Neural Field Training","created":"2023-11-28","cited_by_count":7,"area":null},{"id":"2401.07788","title":"Activations and Gradients Compression for Model-Parallel Training","created":"2024-01-15","cited_by_count":7,"area":null},{"id":"2411.00999","title":"Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers","created":"2024-11-01","cited_by_count":7,"area":null},{"id":"2603.21191","title":"On the Role of Batch Size in Stochastic Conditional Gradient Methods","created":"2026-03-22","cited_by_count":7,"area":null},{"id":"2605.19095","title":"ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models","created":"2026-05-18","cited_by_count":7,"area":null},{"id":"2106.02679","title":"Layered gradient accumulation and modular pipeline parallelism: fast and efficient training of large language models","created":"2021-06-04","cited_by_count":6,"area":null}],"next_cursor":"100"}