{"list":"other","total":58,"rows":[{"id":"1512.03385","title":"Deep Residual Learning for Image Recognition","created":"2015-12-10","cited_by_count":52096},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"1912.01703","title":"PyTorch: An Imperative Style, High-Performance Deep Learning Library","created":"2019-12-03","cited_by_count":20305},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019},{"id":"1909.08053","title":"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism","created":"2019-09-17","cited_by_count":1868},{"id":"1706.02677","title":"Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour","created":"2017-06-08","cited_by_count":1492},{"id":"1609.04836","title":"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima","created":"2016-09-15","cited_by_count":1352},{"id":"1910.02054","title":"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models","created":"2019-10-04","cited_by_count":1301},{"id":"2112.11446","title":"Scaling Language Models: Methods, Analysis & Insights from Training Gopher","created":"2021-12-08","cited_by_count":1202},{"id":"1606.04838","title":"Optimization Methods for Large-Scale Machine Learning","created":"2016-06-15","cited_by_count":1150},{"id":"1309.5549","title":"Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming","created":"2013-09-22","cited_by_count":724},{"id":"1904.00962","title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","created":"2019-04-01","cited_by_count":674},{"id":"2304.11277","title":"PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel","created":"2023-04-21","cited_by_count":660},{"id":"2201.11990","title":"Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model","created":"2022-01-28","cited_by_count":559},{"id":"1905.11881","title":"Why gradient clipping accelerates training: A theoretical justification for adaptivity","created":"2019-05-28","cited_by_count":341},{"id":"1711.00489","title":"Don't Decay the Learning Rate, Increase the Batch Size","created":"2017-11-01","cited_by_count":320},{"id":"1912.02365","title":"Lower Bounds for Non-Convex Stochastic Optimization","created":"2019-12-05","cited_by_count":307},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"1705.08741","title":"Train longer, generalize better: closing the generalization gap in large batch training of neural networks","created":"2017-05-24","cited_by_count":279},{"id":"1705.08292","title":"The Marginal Value of Adaptive Gradient Methods in Machine Learning","created":"2017-05-23","cited_by_count":250},{"id":"1811.03600","title":"Measuring the Effects of Data Parallelism on Neural Network Training","created":"2018-11-08","cited_by_count":193},{"id":"1806.01811","title":"AdaGrad stepsizes: Sharp convergence over nonconvex landscapes","created":"2018-06-05","cited_by_count":182},{"id":"2003.02395","title":"A Simple Convergence Proof of Adam and Adagrad","created":"2020-03-05","cited_by_count":171},{"id":"2002.03329","title":"Better Theory for SGD in the Nonconvex World","created":"2020-02-09","cited_by_count":158},{"id":"1805.08114","title":"On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes","created":"2018-05-21","cited_by_count":156},{"id":"1603.06160","title":"Stochastic Variance Reduction for Nonconvex Optimization","created":"2016-03-19","cited_by_count":145},{"id":"1810.07288","title":"Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron","created":"2018-10-16","cited_by_count":143},{"id":"1710.06451","title":"A Bayesian Perspective on Generalization and Stochastic Gradient Descent","created":"2017-10-17","cited_by_count":135},{"id":"1907.04164","title":"Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model","created":"2019-07-09","cited_by_count":117},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98},{"id":"1808.05671","title":"On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization","created":"2018-08-16","cited_by_count":91},{"id":"2006.15704","title":"PyTorch Distributed: Experiences on Accelerating Data Parallel Training","created":"2020-06-28","cited_by_count":90},{"id":"2010.02519","title":"Improved Analysis of Clipping Algorithms for Non-convex Optimization","created":"2020-10-05","cited_by_count":89},{"id":"2010.00892","title":"Variance-Reduced Methods for Machine Learning","created":"2020-10-02","cited_by_count":79},{"id":"2007.14294","title":"A High Probability Analysis of Adaptive SGD with Momentum","created":"2020-07-28","cited_by_count":61},{"id":"2202.05791","title":"The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance","created":"2022-02-11","cited_by_count":61},{"id":"2305.18471","title":"Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions","created":"2023-05-29","cited_by_count":57},{"id":"2306.00204","title":"Toward Understanding Why Adam Converges Faster Than SGD for Transformers","created":"2023-05-31","cited_by_count":53},{"id":"2006.09092","title":"Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training","created":"2020-06-16","cited_by_count":50},{"id":"2302.14843","title":"High Probability Convergence of Stochastic Gradient Methods","created":"2023-02-28","cited_by_count":49},{"id":"1802.05374","title":"A Progressive Batching L-BFGS Method for Machine Learning","created":"2018-02-15","cited_by_count":45},{"id":"2302.06570","title":"Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGD","created":"2023-02-13","cited_by_count":41},{"id":"2204.02833","title":"High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize","created":"2022-04-06","cited_by_count":40},{"id":"1612.05086","title":"Coupling Adaptive Batch Sizes with Learning Rates","created":"2016-12-15","cited_by_count":30},{"id":"1710.11258","title":"Adaptive Sampling Strategies for Stochastic Optimization","created":"2017-10-30","cited_by_count":30},{"id":"2302.08783","title":"SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variance","created":"2023-02-17","cited_by_count":30},{"id":"1712.02029","title":"AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks","created":"2017-12-06","cited_by_count":29},{"id":"1505.06070","title":"Global convergence rate analysis of unconstrained optimization methods based on probabilistic models","created":"2015-05-22","cited_by_count":27},{"id":"2211.02254","title":"How Does Adaptive Optimization Impact Local Neural Network Geometry?","created":"2022-11-04","cited_by_count":24},{"id":"2007.05105","title":"AdaScale SGD: A User-Friendly Algorithm for Distributed Training","created":"2020-07-09","cited_by_count":16},{"id":"1610.05792","title":"Big Batch SGD: Automated Inference using Adaptive Batch Sizes","created":"2016-10-18","cited_by_count":13},{"id":"2206.00712","title":"An Adaptive Sampling Sequential Quadratic Programming Method for Equality Constrained Stochastic Optimization","created":"2022-06-01","cited_by_count":11},{"id":"2305.01018","title":"An Adaptive Sampling Augmented Lagrangian Method for Stochastic Optimization with Deterministic Constraints","created":"2023-05-01","cited_by_count":5},{"id":"2012.03844","title":"Adaptive sampling strategies for risk-averse stochastic optimization with constraints","created":"2020-12-07","cited_by_count":4},{"id":"2109.12213","title":"Adaptive Sampling Quasi-Newton Methods for Zeroth-Order Stochastic Optimization","created":"2021-09-24","cited_by_count":4},{"id":"2012.15411","title":"Constrained and Composite Optimization via Adaptive Sampling Methods","created":"2020-12-31","cited_by_count":3}],"next_cursor":null}