{"total":139,"rows":[{"id":"2302.06675","title":"Symbolic Discovery of Optimization Algorithms","created":"2023-02-13","cited_by_count":434,"area":null},{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265,"area":null},{"id":"2409.20325","title":"Old Optimizer, New Norm: An Anthology","created":"2024-09-30","cited_by_count":193,"area":null},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114,"area":null},{"id":"2207.14484","title":"Adaptive Gradient Methods at the Edge of Stability","created":"2022-07-29","cited_by_count":101,"area":null},{"id":"2106.09352","title":"Large Scale Private Learning via Low-rank Reparametrization","created":"2021-06-17","cited_by_count":88,"area":null},{"id":"2208.11195","title":"Robustness to Unbounded Smoothness of Generalized SignSGD","created":"2022-08-23","cited_by_count":81,"area":null},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80,"area":null},{"id":"2310.04415","title":"Why Do We Need Weight Decay in Modern Deep Learning?","created":"2023-10-06","cited_by_count":72,"area":null},{"id":"2404.04454","title":"Implicit Bias of AdamW: $\\ell_\\infty$ Norm Constrained Optimization","created":"2024-04-05","cited_by_count":58,"area":null},{"id":"2502.00604","title":"Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective","created":"2025-02-02","cited_by_count":56,"area":null},{"id":"2310.05898","title":"Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts","created":"2023-10-09","cited_by_count":51,"area":null},{"id":"2108.11371","title":"Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization","created":"2021-08-25","cited_by_count":49,"area":null},{"id":"2102.08098","title":"GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training","created":"2021-02-16","cited_by_count":44,"area":null},{"id":"2405.13698","title":"How to set AdamW's weight decay as you scale model and dataset size","created":"2024-05-22","cited_by_count":43,"area":null},{"id":"2407.07972","title":"Deconstructing What Makes a Good Optimizer for Language Models","created":"2024-07-10","cited_by_count":41,"area":null},{"id":"2505.21829","title":"In Search of Adam's Secret Sauce","created":"2025-05-27","cited_by_count":40,"area":null},{"id":"2406.10650","title":"The Implicit Bias of Adam on Separable Data","created":"2024-06-15","cited_by_count":36,"area":null},{"id":"2310.02984","title":"Scaling Laws for Associative Memories","created":"2023-10-04","cited_by_count":35,"area":null},{"id":"2605.20119","title":"Toto 2.0: Time Series Forecasting Enters the Scaling Era","created":"2026-05-19","cited_by_count":30,"area":null},{"id":"2101.11075","title":"Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization","created":"2021-01-26","cited_by_count":28,"area":null},{"id":"2312.14334","title":"DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)","created":"2023-12-21","cited_by_count":28,"area":null},{"id":"2510.09827","title":"An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants","created":"2025-10-10","cited_by_count":28,"area":null},{"id":"2101.05471","title":"Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration","created":"2021-01-14","cited_by_count":25,"area":null},{"id":"2402.03496","title":"Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective","created":"2024-02-05","cited_by_count":25,"area":null},{"id":"2305.12475","title":"Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods","created":"2023-05-21","cited_by_count":23,"area":null},{"id":"2410.08198","title":"Adam Exploits $\\ell_\\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity","created":"2024-10-10","cited_by_count":23,"area":null},{"id":"2405.14578","title":"Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling","created":"2024-05-23","cited_by_count":22,"area":null},{"id":"2303.01500","title":"Dropout Reduces Underfitting","created":"2023-03-02","cited_by_count":21,"area":null},{"id":"2402.07114","title":"Towards Quantifying the Preconditioning Effect of Adam","created":"2024-02-11","cited_by_count":19,"area":null},{"id":"2202.12328","title":"Cutting Some Slack for SGD with Adaptive Polyak Stepsizes","created":"2022-02-24","cited_by_count":18,"area":null},{"id":"2510.25000","title":"What Really Matters in Matrix-Whitening Optimizers?","created":"2025-10-28","cited_by_count":16,"area":null},{"id":"2202.06009","title":"Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam","created":"2022-02-12","cited_by_count":15,"area":null},{"id":"2304.07645","title":"Magnitude Invariant Parametrizations Improve Hypernetwork Learning","created":"2023-04-15","cited_by_count":14,"area":null},{"id":"2602.17080","title":"Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum","created":"2026-02-19","cited_by_count":14,"area":null},{"id":"2301.00944","title":"Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learning","created":"2023-01-03","cited_by_count":12,"area":null},{"id":"2506.01260","title":"Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism","created":"2025-06-02","cited_by_count":12,"area":null},{"id":"2603.09952","title":"On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer","created":"2026-03-10","cited_by_count":12,"area":null},{"id":"2105.13937","title":"Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks","created":"2021-05-28","cited_by_count":11,"area":null},{"id":"2209.06302","title":"Optimization without Backpropagation","created":"2022-09-13","cited_by_count":11,"area":null},{"id":"2502.07274","title":"Forget Forgetting: Continual Learning in a World of Abundant Memory","created":"2025-02-11","cited_by_count":11,"area":null},{"id":"2411.07837","title":"FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training","created":"2024-11-12","cited_by_count":10,"area":null},{"id":"2603.15958","title":"Deriving Hyperparameter Scaling Laws via Modern Optimization Theory","created":"2026-03-16","cited_by_count":10,"area":null},{"id":"2411.01713","title":"Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models","created":"2024-11-03","cited_by_count":9,"area":null},{"id":"2506.04430","title":"Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order","created":"2025-06-04","cited_by_count":9,"area":null},{"id":"2203.17020","title":"Logit Normalization for Long-tail Object Detection","created":"2022-03-31","cited_by_count":8,"area":null},{"id":"2308.06703","title":"Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods","created":"2023-08-13","cited_by_count":8,"area":null},{"id":"2602.09314","title":"Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory","created":"2026-02-10","cited_by_count":8,"area":null},{"id":"2305.13187","title":"SignSVRG: fixing SignSGD via variance reduction","created":"2023-05-22","cited_by_count":7,"area":null},{"id":"2411.12135","title":"Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effects","created":"2024-11-19","cited_by_count":7,"area":null},{"id":"2506.06194","title":"Transformative or Conservative? Conservation laws for ResNets and Transformers","created":"2025-06-06","cited_by_count":7,"area":null},{"id":"2602.03839","title":"Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL","created":"2026-02-03","cited_by_count":7,"area":null},{"id":"2605.10468","title":"Can Muon Fine-tune Adam-Pretrained Models?","created":"2026-05-11","cited_by_count":7,"area":null},{"id":"2106.11514","title":"Rethinking Adam: A Twofold Exponential Moving Average Approach","created":"2021-06-22","cited_by_count":6,"area":null},{"id":"2109.01806","title":"On Faster Convergence of Scaled Sign Gradient Descent","created":"2021-09-04","cited_by_count":6,"area":null},{"id":"2306.15755","title":"Adversarial Backdoor Attack by Naturalistic Data Poisoning on Trajectory Prediction in Autonomous Driving","created":"2023-06-27","cited_by_count":6,"area":null},{"id":"2607.20548","title":"SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales","created":"2026-07-13","cited_by_count":6,"area":null},{"id":"2211.02278","title":"A Deep Learning Approach to Generating Photospheric Vector Magnetograms of Solar Active Regions for SOHO/MDI Using SDO/HMI and BBSO Data","created":"2022-11-04","cited_by_count":5,"area":null},{"id":"2502.17009","title":"Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs","created":"2025-02-24","cited_by_count":5,"area":null},{"id":"2512.00763","title":"Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalance","created":"2025-11-30","cited_by_count":5,"area":null},{"id":"2604.15416","title":"StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models","created":"2026-04-16","cited_by_count":5,"area":null},{"id":"2605.06654","title":"Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less","created":"2026-05-07","cited_by_count":5,"area":null},{"id":"2206.14862","title":"Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks","created":"2022-06-29","cited_by_count":4,"area":null},{"id":"2306.00342","title":"Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks","created":"2023-06-01","cited_by_count":4,"area":null},{"id":"2505.22549","title":"DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models","created":"2025-05-28","cited_by_count":4,"area":null},{"id":"2506.00181","title":"On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach","created":"2025-05-30","cited_by_count":4,"area":null},{"id":"2507.05966","title":"Simple Convergence Proof of Adam From a Sign-like Descent Perspective","created":"2025-07-08","cited_by_count":4,"area":null},{"id":"2510.00823","title":"Non-Euclidean Broximal Point Method: A Blueprint for Geometry-Aware Optimization","created":"2025-10-01","cited_by_count":4,"area":null},{"id":"2602.06880","title":"Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization","created":"2026-02-06","cited_by_count":4,"area":null},{"id":"2106.03763","title":"Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks","created":"2021-06-07","cited_by_count":3,"area":null},{"id":"2211.08583","title":"Empirical Study on Optimizer Selection for Out-of-Distribution Generalization","created":"2022-11-15","cited_by_count":3,"area":null},{"id":"2302.01029","title":"On Suppressing Range of Adaptive Stepsizes of Adam to Improve Generalisation Performance","created":"2023-02-02","cited_by_count":3,"area":null},{"id":"2410.14158","title":"A Mirror Descent Perspective of Smoothed Sign Descent","created":"2024-10-18","cited_by_count":3,"area":null},{"id":"2507.08963","title":"Stochastic Approximation with Block Coordinate Optimal Stepsizes","created":"2025-07-11","cited_by_count":3,"area":null},{"id":"2510.27153","title":"Exploring Landscapes for Better Minima along Valleys","created":"2025-10-31","cited_by_count":3,"area":null},{"id":"2606.09658","title":"Muon Learns More Robust and Transferable Features than Adam","created":"2026-06-08","cited_by_count":3,"area":null},{"id":"2607.21356","title":"Emergent Misalignment Recruits a Pre-existing Persona Subspace","created":"2026-07-23","cited_by_count":3,"area":"alignment"},{"id":"2306.16838","title":"Fast Robust Kernel Regression through Sign Gradient Descent with Early Stopping","created":"2023-06-29","cited_by_count":2,"area":null},{"id":"2310.14963","title":"Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lens","created":"2023-10-23","cited_by_count":2,"area":null},{"id":"2310.15976","title":"Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization","created":"2023-10-24","cited_by_count":2,"area":null},{"id":"2403.16372","title":"SignSGD with Federated Voting","created":"2024-03-25","cited_by_count":2,"area":null},{"id":"2502.10784","title":"Preconditioned Inexact Stochastic ADMM for Deep Model","created":"2025-02-15","cited_by_count":2,"area":null},{"id":"2506.22479","title":"Hindsight-Guided Momentum (HGM) Optimizer: An Approach to Adaptive Learning Rate","created":"2025-06-22","cited_by_count":2,"area":null},{"id":"2510.26303","title":"Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime","created":"2025-10-30","cited_by_count":2,"area":null},{"id":"2601.20769","title":"Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidence","created":"2026-01-28","cited_by_count":2,"area":null},{"id":"2602.03001","title":"Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent","created":"2026-02-03","cited_by_count":2,"area":null},{"id":"2603.03226","title":"Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective","created":"2026-03-03","cited_by_count":2,"area":null},{"id":"2605.06615","title":"When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\\ell_1$-norm Lower Bounds","created":"2026-05-07","cited_by_count":2,"area":null},{"id":"2605.18528","title":"Tight Complexity of Muon under Heavy-Tailed Noise","created":"2026-05-18","cited_by_count":2,"area":null},{"id":"2606.19367","title":"Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics","created":"2026-06-11","cited_by_count":2,"area":null},{"id":"2607.01083","title":"Staleness-Learning Rate Scaling Laws for Asynchronous RLHF","created":"2026-07-01","cited_by_count":2,"area":"alignment"},{"id":"2112.06125","title":"Extending AdamW by Leveraging Its Second Moment and Magnitude","created":"2021-12-09","cited_by_count":1,"area":null},{"id":"2203.13273","title":"A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range","created":"2022-03-24","cited_by_count":1,"area":null},{"id":"2206.02034","title":"A Control Theoretic Framework for Adaptive Gradient Optimizers in Machine Learning","created":"2022-06-04","cited_by_count":1,"area":null},{"id":"2311.02818","title":"Signal Processing Meets SGD: From Momentum to Filter","created":"2023-11-06","cited_by_count":1,"area":null},{"id":"2405.04376","title":"Towards Stability of Parameter-Free Optimization","created":"2024-05-07","cited_by_count":1,"area":null},{"id":"2411.17866","title":"Distributed Sign Momentum with Local Steps for Training Transformers","created":"2024-11-26","cited_by_count":1,"area":null},{"id":"2412.02153","title":"Revisiting the Initial Steps in Adaptive Gradient Descent Optimization","created":"2024-12-03","cited_by_count":1,"area":null},{"id":"2506.05271","title":"Tight analyses of first-order methods with error feedback","created":"2025-06-05","cited_by_count":1,"area":null},{"id":"2507.06464","title":"SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam","created":"2025-07-09","cited_by_count":1,"area":null}],"next_cursor":"100"}