{"total":168,"rows":[{"id":"2103.00065","title":"Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability","created":"2021-02-26","cited_by_count":427,"area":null},{"id":"2208.06677","title":"Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models","created":"2022-08-13","cited_by_count":174,"area":null},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124,"area":null},{"id":"2208.09632","title":"Adam Can Converge Without Any Modification On Update Rules","created":"2022-08-20","cited_by_count":114,"area":null},{"id":"2207.14484","title":"Adaptive Gradient Methods at the Edge of Stability","created":"2022-07-29","cited_by_count":101,"area":null},{"id":"2304.13972","title":"Convergence of Adam Under Relaxed Assumptions","created":"2023-04-27","cited_by_count":99,"area":null},{"id":"2208.11195","title":"Robustness to Unbounded Smoothness of Generalized SignSGD","created":"2022-08-23","cited_by_count":81,"area":null},{"id":"2202.05791","title":"The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance","created":"2022-02-11","cited_by_count":61,"area":null},{"id":"2404.04454","title":"Implicit Bias of AdamW: $\\ell_\\infty$ Norm Constrained Optimization","created":"2024-04-05","cited_by_count":58,"area":null},{"id":"2304.09871","title":"A Theory on Adam Instability in Large-Scale Machine Learning","created":"2023-04-19","cited_by_count":56,"area":null},{"id":"2205.02719","title":"Communication-Efficient Adaptive Federated Learning","created":"2022-05-05","cited_by_count":55,"area":null},{"id":"2306.00204","title":"Toward Understanding Why Adam Converges Faster Than SGD for Transformers","created":"2023-05-31","cited_by_count":53,"area":null},{"id":"2112.03459","title":"A Novel Convergence Analysis for Algorithms of the Adam Family","created":"2021-12-07","cited_by_count":50,"area":null},{"id":"2202.00089","title":"Understanding AdamW through Proximal Methods and Scale-Freeness","created":"2022-01-31","cited_by_count":46,"area":null},{"id":"2509.26030","title":"Muon Outperforms Adam in Tail-End Associative Memory Learning","created":"2025-09-30","cited_by_count":46,"area":null},{"id":"2106.08208","title":"SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients","created":"2021-06-15","cited_by_count":40,"area":null},{"id":"2204.02833","title":"High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize","created":"2022-04-06","cited_by_count":40,"area":null},{"id":"2310.17998","title":"Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity","created":"2023-10-27","cited_by_count":38,"area":null},{"id":"2410.16103","title":"LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics","created":"2024-10-21","cited_by_count":38,"area":null},{"id":"2406.10650","title":"The Implicit Bias of Adam on Separable Data","created":"2024-06-15","cited_by_count":36,"area":null},{"id":"2106.11396","title":"BiAdam: Fast Adaptive Bilevel Optimization Methods","created":"2021-06-21","cited_by_count":33,"area":null},{"id":"2509.15816","title":"On the Convergence of Muon and Beyond","created":"2025-09-19","cited_by_count":31,"area":null},{"id":"2208.09900","title":"Provable Adaptivity of Adam under Non-uniform Smoothness","created":"2022-08-21","cited_by_count":30,"area":null},{"id":"2402.01567","title":"Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise","created":"2024-02-02","cited_by_count":26,"area":null},{"id":"2101.05471","title":"Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration","created":"2021-01-14","cited_by_count":25,"area":null},{"id":"2109.05109","title":"Toward Communication Efficient Adaptive Gradient Method","created":"2021-09-10","cited_by_count":25,"area":null},{"id":"2305.12475","title":"Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods","created":"2023-05-21","cited_by_count":23,"area":null},{"id":"2405.18199","title":"Adam with model exponential moving average is effective for nonconvex optimization","created":"2024-05-28","cited_by_count":23,"area":null},{"id":"2410.08198","title":"Adam Exploits $\\ell_\\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity","created":"2024-10-10","cited_by_count":23,"area":null},{"id":"2110.03891","title":"Does Momentum Change the Implicit Regularization on Separable Data?","created":"2021-10-08","cited_by_count":22,"area":null},{"id":"2303.00565","title":"AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks","created":"2023-03-01","cited_by_count":21,"area":null},{"id":"2402.03982","title":"On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions","created":"2024-02-06","cited_by_count":21,"area":null},{"id":"2205.05632","title":"On Distributed Adaptive Optimization with Gradient Compression","created":"2022-05-11","cited_by_count":19,"area":null},{"id":"2402.07114","title":"Towards Quantifying the Preconditioning Effect of Adam","created":"2024-02-11","cited_by_count":19,"area":null},{"id":"2411.02853","title":"ADOPT: Modified Adam Can Converge with Any $\\beta_2$ with the Optimal Rate","created":"2024-11-05","cited_by_count":18,"area":null},{"id":"2109.03194","title":"On the Convergence of Decentralized Adaptive Gradient Methods","created":"2021-09-07","cited_by_count":17,"area":null},{"id":"2206.00743","title":"Nest Your Adaptive Algorithm for Parameter-Agnostic Nonconvex Minimax Optimization","created":"2022-06-01","cited_by_count":16,"area":null},{"id":"2406.04592","title":"Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimization","created":"2024-06-07","cited_by_count":16,"area":null},{"id":"2202.06009","title":"Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam","created":"2022-02-12","cited_by_count":15,"area":null},{"id":"2301.12379","title":"FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering","created":"2023-01-29","cited_by_count":15,"area":null},{"id":"2407.21078","title":"Convergence rates for the Adam optimizer","created":"2024-07-29","cited_by_count":15,"area":null},{"id":"2307.11782","title":"Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case","created":"2023-07-20","cited_by_count":14,"area":null},{"id":"2404.01436","title":"Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance","created":"2024-04-01","cited_by_count":14,"area":null},{"id":"2405.16397","title":"AdaFisher: Adaptive Second Order Optimization via Fisher Information","created":"2024-05-26","cited_by_count":14,"area":null},{"id":"2104.14840","title":"Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimator","created":"2021-04-30","cited_by_count":13,"area":null},{"id":"2106.16101","title":"AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimization","created":"2021-06-30","cited_by_count":13,"area":null},{"id":"2506.04805","title":"Adaptive Preconditioners Trigger Loss Spikes in Adam","created":"2025-06-05","cited_by_count":13,"area":null},{"id":"2510.12402","title":"Cautious Weight Decay","created":"2025-10-14","cited_by_count":13,"area":null},{"id":"2111.00705","title":"Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimization","created":"2021-11-01","cited_by_count":12,"area":null},{"id":"2105.13937","title":"Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks","created":"2021-05-28","cited_by_count":11,"area":null},{"id":"2205.14473","title":"Efficient-Adam: Communication-Efficient Distributed Adam","created":"2022-05-28","cited_by_count":11,"area":null},{"id":"2410.19964","title":"Understanding Adam Requires Better Rotation Dependent Assumptions","created":"2024-10-25","cited_by_count":11,"area":null},{"id":"2104.12676","title":"Solving a class of non-convex min-max games using adaptive momentum methods","created":"2021-04-26","cited_by_count":10,"area":null},{"id":"2107.08649","title":"Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function","created":"2021-07-19","cited_by_count":10,"area":null},{"id":"2407.18365","title":"FADAS: Towards Federated Adaptive Asynchronous Optimization","created":"2024-07-25","cited_by_count":10,"area":null},{"id":"2109.14213","title":"On the One-sided Convergence of Adam-type Algorithms in Non-convex Non-concave Min-max Optimization","created":"2021-09-29","cited_by_count":9,"area":null},{"id":"2206.06257","title":"Distributed Adversarial Training to Robustify Deep Neural Networks at Scale","created":"2022-06-13","cited_by_count":9,"area":null},{"id":"2210.02070","title":"Non-Convergence and Limit Cycles in the Adam optimizer","created":"2022-10-05","cited_by_count":9,"area":null},{"id":"2102.06527","title":"Mutually exciting point process graphs for modelling dynamic networks","created":"2021-02-11","cited_by_count":8,"area":null},{"id":"2201.11989","title":"Existence and Estimation of Critical Batch Size for Training Generative Adversarial Networks with Two Time-Scale Update Rule","created":"2022-01-28","cited_by_count":8,"area":null},{"id":"2210.17478","title":"TiAda: A Time-scale Adaptive Algorithm for Nonconvex Minimax Optimization","created":"2022-10-31","cited_by_count":8,"area":null},{"id":"2411.19671","title":"On the Performance Analysis of Momentum Method: A Frequency Domain Perspective","created":"2024-11-29","cited_by_count":8,"area":null},{"id":"2504.19426","title":"Sharp higher order convergence rates for the Adam optimizer","created":"2025-04-28","cited_by_count":8,"area":null},{"id":"2505.11840","title":"On the $O(\\frac{\\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\\ell_1$ Norm","created":"2025-05-17","cited_by_count":8,"area":null},{"id":"2102.07314","title":"The Role of Momentum Parameters in the Optimal Convergence of Adaptive Polyak's Heavy-ball Methods","created":"2021-02-15","cited_by_count":7,"area":null},{"id":"2110.01543","title":"Stochastic Anderson Mixing for Nonconvex Stochastic Optimization","created":"2021-10-04","cited_by_count":7,"area":null},{"id":"2201.11204","title":"On the Convergence of mSGD and AdaGrad for Stochastic Optimization","created":"2022-01-26","cited_by_count":7,"area":null},{"id":"2211.01122","title":"Fast Adaptive Federated Bilevel Optimization","created":"2022-11-02","cited_by_count":7,"area":null},{"id":"2403.04236","title":"Regularized DeepIV with Model Selection","created":"2024-03-07","cited_by_count":7,"area":null},{"id":"2106.11514","title":"Rethinking Adam: A Twofold Exponential Moving Average Approach","created":"2021-06-22","cited_by_count":6,"area":null},{"id":"2107.03331","title":"KOALA: A Kalman Optimization Algorithm with Loss Adaptivity","created":"2021-07-07","cited_by_count":6,"area":null},{"id":"2110.04844","title":"Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits","created":"2021-10-10","cited_by_count":6,"area":null},{"id":"2307.00631","title":"Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers","created":"2023-07-02","cited_by_count":6,"area":null},{"id":"2606.04662","title":"Why Muon Outperforms Adam: A Curvature Perspective","created":"2026-06-03","cited_by_count":6,"area":null},{"id":"2102.09804","title":"Local Convergence of Adaptive Gradient Descent Optimizers","created":"2021-02-19","cited_by_count":5,"area":null},{"id":"2110.07435","title":"Adaptive Differentially Private Empirical Risk Minimization","created":"2021-10-14","cited_by_count":5,"area":null},{"id":"2201.05500","title":"Communication-Efficient TeraByte-Scale Model Training Framework for Online Advertising","created":"2022-01-05","cited_by_count":5,"area":null},{"id":"2205.14811","title":"Last-iterate convergence analysis of stochastic momentum methods for neural networks","created":"2022-05-30","cited_by_count":5,"area":null},{"id":"2305.18240","title":"XGrad: Boosting Gradient-Based Optimizers With Weight Prediction","created":"2023-05-26","cited_by_count":5,"area":null},{"id":"2506.01863","title":"Unified Scaling Laws for Compressed Representations","created":"2025-06-02","cited_by_count":5,"area":null},{"id":"2104.14562","title":"Stochastic Mirror Descent for Low-Rank Tensor Decomposition Under Non-Euclidean Losses","created":"2021-04-29","cited_by_count":4,"area":null},{"id":"2105.13650","title":"Data Augmentation for Text Generation Without Any Augmented Data","created":"2021-05-28","cited_by_count":4,"area":null},{"id":"2109.08282","title":"AdaLoss: A computationally-efficient and provably convergent adaptive gradient method","created":"2021-09-17","cited_by_count":4,"area":null},{"id":"2110.05454","title":"Momentum Centering and Asynchronous Update for Adaptive Gradient Methods","created":"2021-10-11","cited_by_count":4,"area":null},{"id":"2204.08306","title":"A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networks","created":"2022-04-18","cited_by_count":4,"area":null},{"id":"2206.13290","title":"Theoretical analysis of Adam using hyperparameters close to one without Lipschitz smoothness","created":"2022-06-27","cited_by_count":4,"area":null},{"id":"2312.01658","title":"AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix","created":"2023-12-04","cited_by_count":4,"area":null},{"id":"2411.05746","title":"Continuous-Time Analysis of Adaptive Optimization and Normalization","created":"2024-11-08","cited_by_count":4,"area":null},{"id":"2507.05966","title":"Simple Convergence Proof of Adam From a Sign-like Descent Perspective","created":"2025-07-08","cited_by_count":4,"area":null},{"id":"2603.09581","title":"Towards Understanding Adam Convergence on Highly Degenerate Polynomials","created":"2026-03-10","cited_by_count":4,"area":null},{"id":"2110.00532","title":"Layer-wise and Dimension-wise Locally Adaptive Federated Learning","created":"2021-10-01","cited_by_count":3,"area":null},{"id":"2201.12678","title":"A Stochastic Bundle Method for Interpolating Networks","created":"2022-01-29","cited_by_count":3,"area":null},{"id":"2203.12191","title":"An Adaptive Gradient Method with Energy and Momentum","created":"2022-03-23","cited_by_count":3,"area":null},{"id":"2211.14292","title":"Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression","created":"2022-11-25","cited_by_count":3,"area":null},{"id":"2301.00631","title":"Stochastic Variable Metric Proximal Gradient with variance reduction for non-convex composite optimization","created":"2023-01-02","cited_by_count":3,"area":null},{"id":"2402.05264","title":"AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size","created":"2024-02-07","cited_by_count":3,"area":null},{"id":"2404.08624","title":"Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networks","created":"2024-04-12","cited_by_count":3,"area":null},{"id":"2409.00029","title":"Attack Anything: Blind DNNs via Universal Background Adversarial Attack","created":"2024-08-17","cited_by_count":3,"area":null},{"id":"2411.04775","title":"Learning dynamical systems from data: Gradient-based dictionary optimization","created":"2024-11-07","cited_by_count":3,"area":null},{"id":"2504.18911","title":"A Langevin sampling algorithm inspired by the Adam optimizer","created":"2025-04-26","cited_by_count":3,"area":null}],"next_cursor":"100"}