{"total":98,"rows":[{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124,"area":null},{"id":"2503.12645","title":"Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization","created":"2025-03-16","cited_by_count":92,"area":null},{"id":"2411.10438","title":"MARS: Unleashing the Power of Variance Reduction for Training Large Models","created":"2024-11-15","cited_by_count":64,"area":null},{"id":"2509.26030","title":"Muon Outperforms Adam in Tail-End Associative Memory Learning","created":"2025-09-30","cited_by_count":46,"area":null},{"id":"2306.01264","title":"Convex and Non-convex Optimization Under Generalized Smoothness","created":"2023-06-02","cited_by_count":44,"area":null},{"id":"2412.19529","title":"Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clipping","created":"2024-12-27","cited_by_count":42,"area":null},{"id":"2310.17998","title":"Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity","created":"2023-10-27","cited_by_count":38,"area":null},{"id":"2410.16103","title":"LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics","created":"2024-10-21","cited_by_count":38,"area":null},{"id":"2305.16284","title":"DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method","created":"2023-05-25","cited_by_count":35,"area":null},{"id":"2509.15816","title":"On the Convergence of Muon and Beyond","created":"2025-09-19","cited_by_count":31,"area":null},{"id":"2311.03252","title":"Parameter-Agnostic Optimization under Relaxed Smoothness","created":"2023-11-06","cited_by_count":29,"area":null},{"id":"2404.02827","title":"BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models","created":"2024-04-03","cited_by_count":27,"area":null},{"id":"2402.01567","title":"Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise","created":"2024-02-02","cited_by_count":26,"area":null},{"id":"2405.18199","title":"Adam with model exponential moving average is effective for nonconvex optimization","created":"2024-05-28","cited_by_count":23,"area":null},{"id":"2410.08198","title":"Adam Exploits $\\ell_\\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity","created":"2024-10-10","cited_by_count":23,"area":null},{"id":"2406.15244","title":"AdaGrad under Anisotropic Smoothness","created":"2024-06-21","cited_by_count":22,"area":null},{"id":"2402.03982","title":"On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions","created":"2024-02-06","cited_by_count":21,"area":null},{"id":"2411.02853","title":"ADOPT: Modified Adam Can Converge with Any $\\beta_2$ with the Optimal Rate","created":"2024-11-05","cited_by_count":18,"area":null},{"id":"2603.28254","title":"MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration","created":"2026-03-30","cited_by_count":17,"area":null},{"id":"2401.09587","title":"Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis","created":"2024-01-17","cited_by_count":15,"area":null},{"id":"2407.08100","title":"Non-convergence of Adam and other adaptive stochastic gradient descent optimization methods for non-vanishing learning rates","created":"2024-07-11","cited_by_count":15,"area":null},{"id":"2407.21078","title":"Convergence rates for the Adam optimizer","created":"2024-07-29","cited_by_count":15,"area":null},{"id":"2307.11782","title":"Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case","created":"2023-07-20","cited_by_count":14,"area":null},{"id":"2403.15146","title":"On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond","created":"2024-03-22","cited_by_count":14,"area":null},{"id":"2404.01436","title":"Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance","created":"2024-04-01","cited_by_count":14,"area":null},{"id":"2502.07923","title":"Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under $(L_0, L_1)$-Smoothness","created":"2025-02-11","cited_by_count":13,"area":null},{"id":"2506.01260","title":"Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism","created":"2025-06-02","cited_by_count":12,"area":null},{"id":"2410.19964","title":"Understanding Adam Requires Better Rotation Dependent Assumptions","created":"2024-10-25","cited_by_count":11,"area":null},{"id":"2411.07724","title":"Convergence Rate Analysis of LION","created":"2024-11-12","cited_by_count":11,"area":null},{"id":"2511.04622","title":"ODE approximation for the Adam algorithm: General and overparametrized setting","created":"2025-11-06","cited_by_count":10,"area":null},{"id":"2409.19212","title":"An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness","created":"2024-09-28","cited_by_count":9,"area":null},{"id":"2410.04458","title":"A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD","created":"2024-10-06","cited_by_count":9,"area":null},{"id":"2311.02000","title":"High Probability Convergence of Adam Under Unbounded Gradients and Affine Variance Noise","created":"2023-11-03","cited_by_count":8,"area":null},{"id":"2412.11768","title":"No More Adam: Learning Rate Scaling at Initialization is All You Need","created":"2024-12-16","cited_by_count":8,"area":null},{"id":"2504.19426","title":"Sharp higher order convergence rates for the Adam optimizer","created":"2025-04-28","cited_by_count":8,"area":null},{"id":"2505.11840","title":"On the $O(\\frac{\\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\\ell_1$ Norm","created":"2025-05-17","cited_by_count":8,"area":null},{"id":"2405.19440","title":"MGDA Converges under Generalized Smoothness, Provably","created":"2024-05-29","cited_by_count":7,"area":null},{"id":"2406.00153","title":"$\\mu$LO: Compute-Efficient Meta-Generalization of Learned Optimizers","created":"2024-05-31","cited_by_count":7,"area":null},{"id":"2412.20017","title":"A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness","created":"2024-12-28","cited_by_count":7,"area":null},{"id":"2606.04662","title":"Why Muon Outperforms Adam: A Curvature Perspective","created":"2026-06-03","cited_by_count":6,"area":null},{"id":"2502.17340","title":"Low-rank bias, weight decay, and model merging in neural networks","created":"2025-02-24","cited_by_count":5,"area":null},{"id":"2505.00347","title":"Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics","created":"2025-05-01","cited_by_count":5,"area":null},{"id":"2410.14054","title":"Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimization","created":"2024-10-17","cited_by_count":4,"area":null},{"id":"2410.16871","title":"Error Feedback under $(L_0,L_1)$-Smoothness: Normalization and Momentum","created":"2024-10-22","cited_by_count":4,"area":null},{"id":"2412.17107","title":"Grams: Gradient Descent with Adaptive Momentum Scaling","created":"2024-12-22","cited_by_count":4,"area":null},{"id":"2506.00181","title":"On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach","created":"2025-05-30","cited_by_count":4,"area":null},{"id":"2507.05966","title":"Simple Convergence Proof of Adam From a Sign-like Descent Perspective","created":"2025-07-08","cited_by_count":4,"area":null},{"id":"2509.07972","title":"Theoretical Analysis on how Learning Rate Warmup Accelerates Convergence","created":"2025-09-09","cited_by_count":4,"area":null},{"id":"2602.08372","title":"Dynamic Regret via Discounted-to-Dynamic Reduction with Applications to Curved Losses and Adam Optimizer","created":"2026-02-09","cited_by_count":4,"area":null},{"id":"2502.18153","title":"SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation","created":"2025-02-25","cited_by_count":3,"area":null},{"id":"2602.01105","title":"OLion: Approaching the Hadamard Ideal by Intersecting Spectral and $\\ell_{\\infty}$ Implicit Biases","created":"2026-02-01","cited_by_count":3,"area":null},{"id":"2603.18899","title":"Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method","created":"2026-03-19","cited_by_count":3,"area":null},{"id":"2606.09658","title":"Muon Learns More Robust and Transferable Features than Adam","created":"2026-06-08","cited_by_count":3,"area":null},{"id":"2406.13041","title":"Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum","created":"2024-06-18","cited_by_count":2,"area":null},{"id":"2411.07120","title":"Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees","created":"2024-11-11","cited_by_count":2,"area":null},{"id":"2411.19647","title":"CAdam: Confidence-Based Optimization for Online Learning","created":"2024-11-29","cited_by_count":2,"area":null},{"id":"2509.15399","title":"Adaptive Algorithms with Sharp Convergence Rates for Stochastic Hierarchical Optimization","created":"2025-09-18","cited_by_count":2,"area":null},{"id":"2510.26303","title":"Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime","created":"2025-10-30","cited_by_count":2,"area":null},{"id":"2605.18528","title":"Tight Complexity of Muon under Heavy-Tailed Noise","created":"2026-05-18","cited_by_count":2,"area":null},{"id":"2606.14560","title":"Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success","created":"2026-06-12","cited_by_count":2,"area":null},{"id":"2606.17526","title":"MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization","created":"2026-06-16","cited_by_count":2,"area":null},{"id":"2402.01621","title":"Stochastic Two Points Method for Deep Model Zeroth-order Optimization","created":"2024-02-02","cited_by_count":1,"area":null},{"id":"2405.04376","title":"Towards Stability of Parameter-Free Optimization","created":"2024-05-07","cited_by_count":1,"area":null},{"id":"2503.03908","title":"On the Convergence of Adam-Type Algorithm for Bilevel Optimization under Unbounded Smoothness","created":"2025-03-05","cited_by_count":1,"area":null},{"id":"2505.15013","title":"Convergence of Adam in Deep ReLU Networks via Directional Complexity and Kakeya Bounds","created":"2025-05-21","cited_by_count":1,"area":null},{"id":"2505.22085","title":"PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning","created":"2025-05-28","cited_by_count":1,"area":null},{"id":"2506.07088","title":"Pointwise confidence estimation in the non-linear $\\ell^2$-regularized least squares","created":"2025-06-08","cited_by_count":1,"area":null},{"id":"2507.06464","title":"SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam","created":"2025-07-09","cited_by_count":1,"area":null},{"id":"2508.06884","title":"Near-Optimal Convergence of Accelerated Gradient Methods under Generalized and $(L_0, L_1)$-Smoothness","created":"2025-08-09","cited_by_count":1,"area":null},{"id":"2601.07326","title":"Convergence Rate Analysis of the AdamW-style Shampoo: Unifying One-Sided and Two-Sided Preconditioning","created":"2026-01-12","cited_by_count":1,"area":null},{"id":"2602.05657","title":"Tight Long-Term Tail Decay of (Clipped) SGD in Non-Convex Optimization","created":"2026-02-05","cited_by_count":1,"area":null},{"id":"2603.10095","title":"Rethinking Adam for Time Series Forecasting: A Simple Heuristic to Improve Optimization under Distribution Shifts","created":"2026-03-10","cited_by_count":1,"area":null},{"id":"2605.04269","title":"Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization","created":"2026-05-05","cited_by_count":1,"area":null},{"id":"2606.23676","title":"Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?","created":"2026-06-22","cited_by_count":1,"area":null},{"id":"2606.28879","title":"Analysis of Adam Algorithms for Stochastic Dynamic Systems","created":"2026-06-27","cited_by_count":1,"area":null},{"id":"2607.04233","title":"Unified convergence analysis for gradient descent optimization methods in the training of deep neural networks","created":"2026-07-05","cited_by_count":1,"area":null},{"id":"2607.14536","title":"Muse: Representation Geometry of Muon Beyond Normalized Momentum","created":"2026-07-16","cited_by_count":1,"area":null},{"id":"2402.00389","title":"On the $O(\\frac{\\sqrt{d}}{T^{1/4}})$ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\\ell_1$ Norm","created":"2024-02-01","cited_by_count":0,"area":null},{"id":"2405.16255","title":"GeoAdaLer: Geometric Insights into Adaptive Stochastic Gradient Descent Algorithms","created":"2024-05-25","cited_by_count":0,"area":null},{"id":"2411.15795","title":"Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization","created":"2024-11-24","cited_by_count":0,"area":null},{"id":"2507.04779","title":"Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks","created":"2025-07-07","cited_by_count":0,"area":null},{"id":"2507.08913","title":"Revisiting Convergence: Shuffling Complexity Beyond Lipschitz Smoothness","created":"2025-07-11","cited_by_count":0,"area":null},{"id":"2509.08726","title":"Decentralized Stochastic Nonconvex Optimization under the $(L_0,L_1)$-Smoothness","created":"2025-09-10","cited_by_count":0,"area":null},{"id":"2511.00469","title":"Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optima","created":"2025-11-01","cited_by_count":0,"area":null},{"id":"2511.20516","title":"Adam Simplified: Bias Correction Debunked","created":"2025-11-25","cited_by_count":0,"area":null},{"id":"2601.03566","title":"Provably Convergent Decentralized Optimization over Directed Graphs under Generalized Smoothness","created":"2026-01-07","cited_by_count":0,"area":null},{"id":"2603.07122","title":"Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers","created":"2026-03-07","cited_by_count":0,"area":null},{"id":"2603.12512","title":"Byzantine-Robust Optimization under $(L_0, L_1)$-Smoothness","created":"2026-03-12","cited_by_count":0,"area":null},{"id":"2604.18492","title":"Barrier-enforced multi-objective optimization for direct point and sharp interval forecasting","created":"2026-04-20","cited_by_count":0,"area":null},{"id":"2605.10335","title":"PowerStep: Memory-Efficient Adaptive Optimization via $\\ell_p$-Norm Steepest Descent","created":"2026-05-11","cited_by_count":0,"area":null},{"id":"2605.15522","title":"Stochastic Non-Smooth Convex Optimization with Unbounded Gradients","created":"2026-05-15","cited_by_count":0,"area":null},{"id":"2605.19392","title":"Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approach","created":"2026-05-19","cited_by_count":0,"area":null},{"id":"2605.21177","title":"ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning","created":"2026-05-20","cited_by_count":0,"area":null},{"id":"2606.22326","title":"Adam Converges in Nonsmooth Nonconvex Optimization","created":"2026-06-21","cited_by_count":0,"area":null},{"id":"2608.30382","title":"Convergence rates for the RMSprop optimizer with full control of the hyperparameters","created":"2026-08-31","cited_by_count":0,"area":null},{"id":"2609.30276","title":"Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothness","created":"2026-08-12","cited_by_count":0,"area":null},{"id":"2609.37787","title":"Adam under Generalized Smoothness with Second-Moment-Type Stochastic Gradients","created":"2026-09-29","cited_by_count":0,"area":null},{"id":"2610.16923","title":"Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization","created":"2026-10-12","cited_by_count":0,"area":null}],"next_cursor":null}