{"list":"other","total":62,"rows":[{"id":"2302.13971","title":"LLaMA: Open and Efficient Foundation Language Models","created":"2023-02-27","cited_by_count":14132},{"id":"2307.09288","title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","created":"2023-07-18","cited_by_count":13028},{"id":"1608.03983","title":"SGDR: Stochastic Gradient Descent with Warm Restarts","created":"2016-08-13","cited_by_count":5045},{"id":"1606.08415","title":"Gaussian Error Linear Units (GELUs)","created":"2016-06-27","cited_by_count":3502},{"id":"1904.09237","title":"On the Convergence of Adam and Beyond","created":"2019-04-19","cited_by_count":841},{"id":"1309.5549","title":"Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming","created":"2013-09-22","cited_by_count":724},{"id":"1904.00962","title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","created":"2019-04-01","cited_by_count":674},{"id":"1802.04434","title":"signSGD: Compressed Optimisation for Non-Convex Problems","created":"2018-02-13","cited_by_count":571},{"id":"1708.03888","title":"Large Batch Training of Convolutional Networks","created":"2017-08-13","cited_by_count":535},{"id":"1912.13213","title":"Online Learning: A Modern Introduction Using Convex Optimization","created":"2019-12-31","cited_by_count":417},{"id":"2302.09419","title":"A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT","created":"2023-02-18","cited_by_count":368},{"id":"1905.11881","title":"Why gradient clipping accelerates training: A theoretical justification for adaptivity","created":"2019-05-28","cited_by_count":341},{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265},{"id":"1912.09789","title":"A Survey on Distributed Machine Learning","created":"2019-12-20","cited_by_count":188},{"id":"1806.01811","title":"AdaGrad stepsizes: Sharp convergence over nonconvex landscapes","created":"2018-06-05","cited_by_count":182},{"id":"2003.02395","title":"A Simple Convergence Proof of Adam and Adagrad","created":"2020-03-05","cited_by_count":171},{"id":"2301.11235","title":"Handbook of Convergence Theorems for (Stochastic) Gradient Methods","created":"2023-01-26","cited_by_count":163},{"id":"1611.07476","title":"Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond","created":"2016-11-22","cited_by_count":159},{"id":"1805.08114","title":"On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes","created":"2018-05-21","cited_by_count":156},{"id":"2006.15429","title":"Understanding Gradient Clipping in Private SGD: A Geometric Perspective","created":"2020-06-27","cited_by_count":142},{"id":"1603.05953","title":"Katyusha: The First Direct Acceleration of Stochastic Gradient Methods","created":"2016-03-18","cited_by_count":135},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124},{"id":"2208.09632","title":"Adam Can Converge Without Any Modification On Update Rules","created":"2022-08-20","cited_by_count":114},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114},{"id":"2005.10785","title":"Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping","created":"2020-05-21","cited_by_count":104},{"id":"1910.05446","title":"On Empirical Comparisons of Optimizers for Deep Learning","created":"2019-10-11","cited_by_count":103},{"id":"2304.13972","title":"Convergence of Adam Under Relaxed Assumptions","created":"2023-04-27","cited_by_count":99},{"id":"2010.02519","title":"Improved Analysis of Clipping Algorithms for Non-convex Optimization","created":"2020-10-05","cited_by_count":89},{"id":"1601.01974","title":"Scale-Free Online Learning","created":"2016-01-08","cited_by_count":85},{"id":"2305.01588","title":"Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees","created":"2023-05-02","cited_by_count":83},{"id":"2208.11195","title":"Robustness to Unbounded Smoothness of Generalized SignSGD","created":"2022-08-23","cited_by_count":81},{"id":"1904.12838","title":"The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares","created":"2019-04-29","cited_by_count":68},{"id":"1602.05419","title":"Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression","created":"2016-02-17","cited_by_count":64},{"id":"2202.05791","title":"The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance","created":"2022-02-11","cited_by_count":61},{"id":"2305.18471","title":"Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions","created":"2023-05-29","cited_by_count":57},{"id":"1809.02864","title":"Online Adaptive Methods, Universality and Acceleration","created":"2018-09-08","cited_by_count":51},{"id":"1706.05507","title":"Variants of RMSProp and Adagrad with Logarithmic Regret Bounds","created":"2017-06-17","cited_by_count":50},{"id":"2112.03459","title":"A Novel Convergence Analysis for Algorithms of the Adam Family","created":"2021-12-07","cited_by_count":50},{"id":"2302.14843","title":"High Probability Convergence of Stochastic Gradient Methods","created":"2023-02-28","cited_by_count":49},{"id":"2202.00089","title":"Understanding AdamW through Proximal Methods and Scale-Freeness","created":"2022-01-31","cited_by_count":46},{"id":"2010.04261","title":"Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks","created":"2020-10-08","cited_by_count":45},{"id":"2302.06570","title":"Beyond Uniform Smoothness: A Stopped Analysis of Adaptive SGD","created":"2023-02-13","cited_by_count":41},{"id":"2204.02833","title":"High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize","created":"2022-04-06","cited_by_count":40},{"id":"1807.06766","title":"Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration","created":"2018-07-18","cited_by_count":39},{"id":"2310.17998","title":"Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity","created":"2023-10-27","cited_by_count":38},{"id":"1502.05744","title":"Scale-Free Algorithms for Online Linear Optimization","created":"2015-02-19","cited_by_count":30},{"id":"2302.08783","title":"SGD with AdaGrad Stepsizes: Full Adaptivity with High Probability to Unknown Parameters, Unbounded Gradients and Affine Variance","created":"2023-02-17","cited_by_count":30},{"id":"1803.02865","title":"WNGrad: Learn the Learning Rate in Gradient Descent","created":"2018-03-07","cited_by_count":24},{"id":"2402.03982","title":"On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions","created":"2024-02-06","cited_by_count":21},{"id":"2402.07114","title":"Towards Quantifying the Preconditioning Effect of Adam","created":"2024-02-11","cited_by_count":19},{"id":"1810.04723","title":"Tight Dimension Independent Lower Bound on the Expected Convergence Rate for Diminishing Step Sizes in SGD","created":"2018-10-10","cited_by_count":18},{"id":"2008.01805","title":"Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry","created":"2020-08-04","cited_by_count":17},{"id":"2406.04592","title":"Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimization","created":"2024-06-07","cited_by_count":16},{"id":"2006.06835","title":"Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search)","created":"2020-06-11","cited_by_count":15},{"id":"2110.14109","title":"Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums","created":"2021-10-27","cited_by_count":15},{"id":"2402.13794","title":"Revisiting Convergence of AdaGrad with Relaxed Assumptions","created":"2024-02-21","cited_by_count":15},{"id":"1304.4523","title":"Origins of power-law degree distribution in the heterogeneity of human activity in social networks","created":"2013-04-16","cited_by_count":9},{"id":"2312.14567","title":"Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise","created":"2023-12-22","cited_by_count":9},{"id":"2209.00809","title":"Optimal Diagonal Preconditioning","created":"2022-09-02","cited_by_count":7},{"id":"2211.17201","title":"ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT","created":"2022-11-30","cited_by_count":7},{"id":"2110.04844","title":"Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits","created":"2021-10-10","cited_by_count":6},{"id":"1709.05011","title":"ImageNet Training in Minutes","created":"2017-09-14","cited_by_count":4}],"next_cursor":null}