{"total":114,"rows":[{"id":"2406.16793","title":"Adam-mini: Use Fewer Learning Rates To Gain More","created":"2024-06-24","cited_by_count":137,"area":null},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124,"area":null},{"id":"2402.01868","title":"Challenges in Training PINNs: A Loss Landscape Perspective","created":"2024-02-02","cited_by_count":113,"area":null},{"id":"2304.13972","title":"Convergence of Adam Under Relaxed Assumptions","created":"2023-04-27","cited_by_count":99,"area":null},{"id":"2406.19146","title":"Resolving Discrepancies in Compute-Optimal Scaling of Language Models","created":"2024-06-27","cited_by_count":88,"area":null},{"id":"2305.01588","title":"Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees","created":"2023-05-02","cited_by_count":83,"area":null},{"id":"2410.21676","title":"How Does Critical Batch Size Scale in Pre-training?","created":"2024-10-29","cited_by_count":73,"area":null},{"id":"2411.10438","title":"MARS: Unleashing the Power of Variance Reduction for Training Large Models","created":"2024-11-15","cited_by_count":64,"area":null},{"id":"2404.04454","title":"Implicit Bias of AdamW: $\\ell_\\infty$ Norm Constrained Optimization","created":"2024-04-05","cited_by_count":58,"area":null},{"id":"2509.26030","title":"Muon Outperforms Adam in Tail-End Associative Memory Learning","created":"2025-09-30","cited_by_count":46,"area":null},{"id":"2507.07101","title":"Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful","created":"2025-07-09","cited_by_count":41,"area":null},{"id":"2505.21829","title":"In Search of Adam's Secret Sauce","created":"2025-05-27","cited_by_count":40,"area":null},{"id":"2310.17998","title":"Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity","created":"2023-10-27","cited_by_count":38,"area":null},{"id":"2410.16103","title":"LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics","created":"2024-10-21","cited_by_count":38,"area":null},{"id":"2406.10650","title":"The Implicit Bias of Adam on Separable Data","created":"2024-06-15","cited_by_count":36,"area":null},{"id":"2404.02827","title":"BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models","created":"2024-04-03","cited_by_count":27,"area":null},{"id":"2302.01172","title":"STEP: Learning N:M Structured Sparsity Masks from Scratch with Precondition","created":"2023-02-02","cited_by_count":26,"area":null},{"id":"2402.01567","title":"Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise","created":"2024-02-02","cited_by_count":26,"area":null},{"id":"2412.00383","title":"Unified Parameter-Efficient Unlearning for LLMs","created":"2024-11-30","cited_by_count":25,"area":"lr"},{"id":"2305.12475","title":"Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods","created":"2023-05-21","cited_by_count":23,"area":null},{"id":"2405.18199","title":"Adam with model exponential moving average is effective for nonconvex optimization","created":"2024-05-28","cited_by_count":23,"area":null},{"id":"2410.08198","title":"Adam Exploits $\\ell_\\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity","created":"2024-10-10","cited_by_count":23,"area":null},{"id":"2305.03938","title":"Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees","created":"2023-05-06","cited_by_count":22,"area":null},{"id":"2305.15349","title":"On the Convergence of Black-Box Variational Inference","created":"2023-05-24","cited_by_count":22,"area":null},{"id":"2406.15244","title":"AdaGrad under Anisotropic Smoothness","created":"2024-06-21","cited_by_count":22,"area":null},{"id":"2402.03982","title":"On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions","created":"2024-02-06","cited_by_count":21,"area":null},{"id":"2402.07114","title":"Towards Quantifying the Preconditioning Effect of Adam","created":"2024-02-11","cited_by_count":19,"area":null},{"id":"2411.02853","title":"ADOPT: Modified Adam Can Converge with Any $\\beta_2$ with the Optimal Rate","created":"2024-11-05","cited_by_count":18,"area":null},{"id":"2407.08100","title":"Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates","created":"2024-07-11","cited_by_count":15,"area":null},{"id":"2407.21078","title":"Convergence rates for the Adam optimizer","created":"2024-07-29","cited_by_count":15,"area":null},{"id":"2307.11782","title":"Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case","created":"2023-07-20","cited_by_count":14,"area":null},{"id":"2403.15146","title":"On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond","created":"2024-03-22","cited_by_count":14,"area":null},{"id":"2404.01436","title":"Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance","created":"2024-04-01","cited_by_count":14,"area":null},{"id":"2407.20999","title":"MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning","created":"2024-07-30","cited_by_count":14,"area":null},{"id":"2506.04805","title":"Adaptive Preconditioners Trigger Loss Spikes in Adam","created":"2025-06-05","cited_by_count":13,"area":null},{"id":"2506.16659","title":"Memory-Efficient LLM Pretraining via Minimalist Optimizer Design","created":"2025-06-20","cited_by_count":13,"area":null},{"id":"2410.19964","title":"Understanding Adam Requires Better Rotation Dependent Assumptions","created":"2024-10-25","cited_by_count":11,"area":null},{"id":"2411.07724","title":"Convergence Rate Analysis of LION","created":"2024-11-12","cited_by_count":11,"area":null},{"id":"2602.16340","title":"The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks","created":"2026-02-18","cited_by_count":11,"area":null},{"id":"2511.04622","title":"ODE approximation for the Adam algorithm: General and overparametrized setting","created":"2025-11-06","cited_by_count":10,"area":null},{"id":"2410.04458","title":"A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD","created":"2024-10-06","cited_by_count":9,"area":null},{"id":"2210.05607","title":"Divergence Results and Convergence of a Variance Reduced Version of ADAM","created":"2022-10-11","cited_by_count":8,"area":null},{"id":"2210.17478","title":"TiAda: A Time-scale Adaptive Algorithm for Nonconvex Minimax Optimization","created":"2022-10-31","cited_by_count":8,"area":null},{"id":"2311.02000","title":"High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noise","created":"2023-11-03","cited_by_count":8,"area":null},{"id":"2504.19426","title":"Sharp higher order convergence rates for the Adam optimizer","created":"2025-04-28","cited_by_count":8,"area":null},{"id":"2505.11840","title":"On the $O(\\frac{\\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\\ell_1$ Norm","created":"2025-05-17","cited_by_count":8,"area":null},{"id":"2411.00999","title":"Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers","created":"2024-11-01","cited_by_count":7,"area":null},{"id":"2303.10472","title":"Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference","created":"2023-03-18","cited_by_count":6,"area":null},{"id":"2406.11257","title":"ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking","created":"2024-06-17","cited_by_count":6,"area":null},{"id":"2407.04480","title":"LoCo: Low-Bit Communication Adaptor for Large-scale Model Training","created":"2024-07-05","cited_by_count":6,"area":null},{"id":"2606.04662","title":"Why Muon Outperforms Adam: A Curvature Perspective","created":"2026-06-03","cited_by_count":6,"area":null},{"id":"2310.08858","title":"Adam-family Methods with Decoupled Weight Decay in Deep Learning","created":"2023-10-13","cited_by_count":5,"area":null},{"id":"2505.22549","title":"DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models","created":"2025-05-28","cited_by_count":4,"area":null},{"id":"2507.05966","title":"Simple Convergence Proof of Adam From a Sign-like Descent Perspective","created":"2025-07-08","cited_by_count":4,"area":null},{"id":"2602.08372","title":"Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizer","created":"2026-02-09","cited_by_count":4,"area":null},{"id":"2603.09581","title":"Towards Understanding Adam Convergence on Highly Degenerate Polynomials","created":"2026-03-10","cited_by_count":4,"area":null},{"id":"2605.06152","title":"Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes","created":"2026-05-07","cited_by_count":4,"area":null},{"id":"2307.10053","title":"Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization","created":"2023-07-19","cited_by_count":3,"area":null},{"id":"2505.24399","title":"LightSAM: Parameter-Agnostic Sharpness-Aware Minimization","created":"2025-05-30","cited_by_count":3,"area":null},{"id":"2507.08963","title":"Stochastic Approximation with Block Coordinate Optimal Stepsizes","created":"2025-07-11","cited_by_count":3,"area":null},{"id":"2603.18899","title":"Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method","created":"2026-03-19","cited_by_count":3,"area":null},{"id":"2605.18694","title":"Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad","created":"2026-05-18","cited_by_count":3,"area":null},{"id":"2606.09658","title":"Muon Learns More Robust and Transferable Features than Adam","created":"2026-06-08","cited_by_count":3,"area":null},{"id":"2305.05675","title":"UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization","created":"2023-05-09","cited_by_count":2,"area":null},{"id":"2308.00720","title":"Divergence of the ADAM algorithm with fixed-stepsize: a (very) simple example","created":"2023-08-01","cited_by_count":2,"area":null},{"id":"2401.08426","title":"GD doesn't make the cut: Three ways that non-differentiability affects neural network training","created":"2024-01-16","cited_by_count":2,"area":null},{"id":"2411.19647","title":"CAdam: Confidence-Based Optimization for Online Learning","created":"2024-11-29","cited_by_count":2,"area":null},{"id":"2505.17300","title":"Statistical Inference for Online Algorithms","created":"2025-05-22","cited_by_count":2,"area":null},{"id":"2506.22479","title":"Hindsight-Guided Momentum (HGM) Optimizer: An Approach to Adaptive Learning Rate","created":"2025-06-22","cited_by_count":2,"area":null},{"id":"2510.03478","title":"How to Set $\\beta_1, \\beta_2$ in Adam: An Online Learning Perspective","created":"2025-10-03","cited_by_count":2,"area":null},{"id":"2510.26303","title":"Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime","created":"2025-10-30","cited_by_count":2,"area":null},{"id":"2604.17423","title":"A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muon","created":"2026-04-19","cited_by_count":2,"area":null},{"id":"2605.00650","title":"AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments","created":"2026-05-01","cited_by_count":2,"area":null},{"id":"2605.18528","title":"Tight Complexity of Muon under Heavy-Tailed Noise","created":"2026-05-18","cited_by_count":2,"area":null},{"id":"2606.17526","title":"MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization","created":"2026-06-16","cited_by_count":2,"area":null},{"id":"2312.10490","title":"Spatial Deep Learning for Site-Specific Movement Optimization of Aerial Base Stations","created":"2023-12-16","cited_by_count":1,"area":null},{"id":"2406.17954","title":"Why Line Search when you can Plane Search? SO-Friendly Neural Networks allow Per-Iteration Optimization of Learning and Momentum Rates for Every Layer","created":"2024-06-25","cited_by_count":1,"area":null},{"id":"2503.03908","title":"On the Convergence of Adam-Type Algorithm for Bilevel Optimization under Unbounded Smoothness","created":"2025-03-05","cited_by_count":1,"area":null},{"id":"2505.13954","title":"VAMO: Efficient Zeroth-Order Variance Reduction for SGD with Faster Convergence","created":"2025-05-20","cited_by_count":1,"area":null},{"id":"2507.06464","title":"SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam","created":"2025-07-09","cited_by_count":1,"area":null},{"id":"2512.06795","title":"ADAM Optimization with Adaptive Batch Selection","created":"2025-12-07","cited_by_count":1,"area":null},{"id":"2601.07326","title":"Convergence Rate Analysis of the AdamW-style Shampoo: Unifying One-Sided and Two-Sided Preconditioning","created":"2026-01-12","cited_by_count":1,"area":null},{"id":"2602.07632","title":"Scalable Mean-Field Variational Inference via Preconditioned Primal-Dual Optimization","created":"2026-02-07","cited_by_count":1,"area":null},{"id":"2603.02649","title":"HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization","created":"2026-03-03","cited_by_count":1,"area":null},{"id":"2604.08742","title":"Deterministic Adam-Inspired Methods with Accelerated Convergence Rate","created":"2026-04-09","cited_by_count":1,"area":null},{"id":"2606.15702","title":"Schattor: Schatten-family methods for deep learning optimization","created":"2026-06-14","cited_by_count":1,"area":null},{"id":"2606.21433","title":"Central limit theorem for the averaged Adam optimizer","created":"2026-06-19","cited_by_count":1,"area":null},{"id":"2606.23676","title":"Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?","created":"2026-06-22","cited_by_count":1,"area":null},{"id":"2606.28879","title":"Analysis of Adam Algorithms for Stochastic Dynamic Systems","created":"2026-06-27","cited_by_count":1,"area":null},{"id":"2607.03519","title":"On the Convergence of Adam, Revisited","created":"2026-07-03","cited_by_count":1,"area":null},{"id":"2607.04233","title":"Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks","created":"2026-07-05","cited_by_count":1,"area":null},{"id":"2608.15824","title":"Second-Moment Memory in Coordinatewise Adam","created":"2026-08-16","cited_by_count":1,"area":null},{"id":"2608.20638","title":"Adam at the Edge of Stability: Adaptive Feedback, Provable Oscillation, and Gradient Reversal","created":"2026-08-21","cited_by_count":1,"area":null},{"id":"2212.01848","title":"Convergence of ease-controlled Random Reshuffling gradient Algorithms under Lipschitz smoothness","created":"2022-12-04","cited_by_count":0,"area":null},{"id":"2312.02182","title":"Adam-like Algorithm with Smooth Clipping Attains Global Minima: Analysis Based on Ergodicity of Functional SDEs","created":"2023-11-29","cited_by_count":0,"area":null},{"id":"2402.00389","title":"On the $O(\\frac{\\sqrt{d}}{T^{1/4}})$ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\\ell_1$ Norm","created":"2024-02-01","cited_by_count":0,"area":null},{"id":"2405.16255","title":"GeoAdaLer: Geometric Insights into Adaptive Stochastic Gradient Descent Algorithms","created":"2024-05-25","cited_by_count":0,"area":null},{"id":"2411.15795","title":"Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization","created":"2024-11-24","cited_by_count":0,"area":null},{"id":"2503.02147","title":"Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks","created":"2025-03-04","cited_by_count":0,"area":null},{"id":"2507.04779","title":"Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks","created":"2025-07-07","cited_by_count":0,"area":null}],"next_cursor":"100"}