{"total":114,"rows":[{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265,"area":null},{"id":"2406.16793","title":"Adam-mini: Use Fewer Learning Rates To Gain More","created":"2024-06-24","cited_by_count":137,"area":null},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124,"area":null},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98,"area":null},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80,"area":null},{"id":"2404.04454","title":"Implicit Bias of AdamW: $\\ell_\\infty$ Norm Constrained Optimization","created":"2024-04-05","cited_by_count":58,"area":null},{"id":"2503.20762","title":"ASGO: Adaptive Structured Gradient Optimization","created":"2025-03-26","cited_by_count":57,"area":null},{"id":"2310.05898","title":"Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts","created":"2023-10-09","cited_by_count":51,"area":null},{"id":"2407.07972","title":"Deconstructing What Makes a Good Optimizer for Language Models","created":"2024-07-10","cited_by_count":41,"area":null},{"id":"2506.04192","title":"Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise","created":"2025-06-04","cited_by_count":41,"area":null},{"id":"2505.21829","title":"In Search of Adam's Secret Sauce","created":"2025-05-27","cited_by_count":40,"area":null},{"id":"2405.16002","title":"Does SGD really happen in tiny subspaces?","created":"2024-05-25","cited_by_count":36,"area":null},{"id":"2406.10650","title":"The Implicit Bias of Adam on Separable Data","created":"2024-06-15","cited_by_count":36,"area":null},{"id":"2305.16284","title":"DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method","created":"2023-05-25","cited_by_count":35,"area":null},{"id":"2310.02984","title":"Scaling Laws for Associative Memories","created":"2023-10-04","cited_by_count":35,"area":null},{"id":"2510.16981","title":"MuonBP: Faster Muon via Block-Periodic Orthogonalization","created":"2025-10-19","cited_by_count":35,"area":null},{"id":"2501.18965","title":"The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training","created":"2025-01-31","cited_by_count":33,"area":null},{"id":"2402.05738","title":"Implicit Bias and Fast Convergence Rates for Self-attention","created":"2024-02-08","cited_by_count":31,"area":null},{"id":"2402.01567","title":"Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise","created":"2024-02-02","cited_by_count":26,"area":null},{"id":"2402.03496","title":"Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective","created":"2024-02-05","cited_by_count":25,"area":null},{"id":"2505.19227","title":"Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law","created":"2025-05-25","cited_by_count":24,"area":null},{"id":"2405.18199","title":"Adam with model exponential moving average is effective for nonconvex optimization","created":"2024-05-28","cited_by_count":23,"area":null},{"id":"2410.08198","title":"Adam Exploits $\\ell_\\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity","created":"2024-10-10","cited_by_count":23,"area":null},{"id":"2305.15349","title":"On the Convergence of Black-Box Variational Inference","created":"2023-05-24","cited_by_count":22,"area":null},{"id":"2406.15244","title":"AdaGrad under Anisotropic Smoothness","created":"2024-06-21","cited_by_count":22,"area":null},{"id":"2402.03982","title":"On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions","created":"2024-02-06","cited_by_count":21,"area":null},{"id":"2412.13148","title":"SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training","created":"2024-12-17","cited_by_count":20,"area":null},{"id":"2306.09000","title":"When and Why Momentum Accelerates SGD:An Empirical Study","created":"2023-06-15","cited_by_count":19,"area":null},{"id":"2505.24749","title":"SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training","created":"2025-05-30","cited_by_count":19,"area":null},{"id":"2405.12807","title":"FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information","created":"2024-05-21","cited_by_count":18,"area":null},{"id":"2410.22069","title":"Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks","created":"2024-10-29","cited_by_count":18,"area":null},{"id":"2306.12747","title":"Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models","created":"2023-06-22","cited_by_count":15,"area":null},{"id":"2506.12543","title":"Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling","created":"2025-06-14","cited_by_count":15,"area":null},{"id":"2311.04163","title":"Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization","created":"2023-11-07","cited_by_count":13,"area":null},{"id":"2506.16659","title":"Memory-Efficient LLM Pretraining via Minimalist Optimizer Design","created":"2025-06-20","cited_by_count":13,"area":null},{"id":"2508.15706","title":"Overcoming the Communication-Performance Tradeoff in LLM Pretraining","created":"2025-08-21","cited_by_count":12,"area":null},{"id":"2410.19964","title":"Understanding Adam Requires Better Rotation Dependent Assumptions","created":"2024-10-25","cited_by_count":11,"area":null},{"id":"2411.07724","title":"Convergence Rate Analysis of LION","created":"2024-11-12","cited_by_count":11,"area":null},{"id":"2411.07837","title":"FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training","created":"2024-11-12","cited_by_count":10,"area":null},{"id":"2310.04627","title":"Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuning","created":"2023-10-06","cited_by_count":9,"area":null},{"id":"2406.04657","title":"From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks","created":"2024-06-07","cited_by_count":9,"area":null},{"id":"2502.06742","title":"Gradient Multi-Normalization for Stateless and Scalable LLM Training","created":"2025-02-10","cited_by_count":9,"area":null},{"id":"2402.11215","title":"AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods","created":"2024-02-17","cited_by_count":8,"area":null},{"id":"2412.11768","title":"No More Adam: Learning Rate Scaling at Initialization is All You Need","created":"2024-12-16","cited_by_count":8,"area":null},{"id":"2602.09314","title":"Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory","created":"2026-02-10","cited_by_count":8,"area":null},{"id":"2411.00999","title":"Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers","created":"2024-11-01","cited_by_count":7,"area":null},{"id":"2411.12135","title":"Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effects","created":"2024-11-19","cited_by_count":7,"area":null},{"id":"2510.19376","title":"Optimization Benchmark for Diffusion Models on Dynamical Systems","created":"2025-10-22","cited_by_count":7,"area":"diffusion"},{"id":"2602.07729","title":"Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs","created":"2026-02-07","cited_by_count":7,"area":null},{"id":"2502.02132","title":"How Memory in Optimization Algorithms Implicitly Modifies the Loss","created":"2025-02-04","cited_by_count":6,"area":null},{"id":"2505.15624","title":"Mechanistic Insights into Grokking from the Embedding Layer","created":"2025-05-21","cited_by_count":6,"area":null},{"id":"2507.12091","title":"Better Convergence Guarantees for Sign-Based Momentum Methods","created":"2025-07-16","cited_by_count":6,"area":null},{"id":"2509.14230","title":"NIRVANA: Structured Pruning Reimagined for Large Language Model Compression","created":"2025-09-17","cited_by_count":6,"area":null},{"id":"2605.10797","title":"Muown: Row-Norm Control for Muon Optimization","created":"2026-05-11","cited_by_count":6,"area":null},{"id":"2509.24218","title":"Conda: Column-Normalized Adam for Training Large Language Models Faster","created":"2025-09-29","cited_by_count":5,"area":null},{"id":"2512.00763","title":"Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalance","created":"2025-11-30","cited_by_count":5,"area":null},{"id":"2602.18308","title":"JPmHC Dynamical Isometry via Orthogonal Hyper-Connections","created":"2026-02-20","cited_by_count":5,"area":null},{"id":"2604.15416","title":"StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models","created":"2026-04-16","cited_by_count":5,"area":null},{"id":"2605.06654","title":"Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less","created":"2026-05-07","cited_by_count":5,"area":null},{"id":"2412.21124","title":"Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism","created":"2024-12-30","cited_by_count":4,"area":null},{"id":"2505.22549","title":"DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models","created":"2025-05-28","cited_by_count":4,"area":null},{"id":"2507.05966","title":"Simple Convergence Proof of Adam From a Sign-like Descent Perspective","created":"2025-07-08","cited_by_count":4,"area":null},{"id":"2510.00823","title":"Non-Euclidean Broximal Point Method: A Blueprint for Geometry-Aware Optimization","created":"2025-10-01","cited_by_count":4,"area":null},{"id":"2602.01642","title":"The Effect of Mini-Batch Noise on the Implicit Bias of Adam","created":"2026-02-02","cited_by_count":4,"area":null},{"id":"2603.09581","title":"Towards Understanding Adam Convergence on Highly Degenerate Polynomials","created":"2026-03-10","cited_by_count":4,"area":null},{"id":"2410.14158","title":"A Mirror Descent Perspective of Smoothed Sign Descent","created":"2024-10-18","cited_by_count":3,"area":null},{"id":"2510.09103","title":"AdaPM: a Partial Momentum Algorithm for LLM Training","created":"2025-10-10","cited_by_count":3,"area":null},{"id":"2602.00334","title":"Adaptive Momentum and Nonlinear Damping for Neural Network Training","created":"2026-01-30","cited_by_count":3,"area":null},{"id":"2605.11172","title":"Optimistic Dual Averaging Unifies Modern Optimizers","created":"2026-05-11","cited_by_count":3,"area":null},{"id":"2605.18898","title":"A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions","created":"2026-05-17","cited_by_count":3,"area":null},{"id":"2606.09658","title":"Muon Learns More Robust and Transferable Features than Adam","created":"2026-06-08","cited_by_count":3,"area":null},{"id":"2310.15976","title":"Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimization","created":"2023-10-24","cited_by_count":2,"area":null},{"id":"2311.08745","title":"Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization","created":"2023-11-15","cited_by_count":2,"area":null},{"id":"2405.01731","title":"Dynamic Anisotropic Smoothing for Noisy Derivative-Free Optimization","created":"2024-05-02","cited_by_count":2,"area":null},{"id":"2405.17377","title":"How Do the Architecture and Optimizer Affect Representation Learning? On the Training Dynamics of Representations in Deep Neural Networks","created":"2024-05-27","cited_by_count":2,"area":null},{"id":"2510.00236","title":"Per-example gradients: a new frontier for understanding and improving optimizers","created":"2025-09-30","cited_by_count":2,"area":null},{"id":"2510.13680","title":"Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise","created":"2025-10-15","cited_by_count":2,"area":null},{"id":"2510.26303","title":"Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime","created":"2025-10-30","cited_by_count":2,"area":null},{"id":"2605.00650","title":"AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments","created":"2026-05-01","cited_by_count":2,"area":null},{"id":"2605.06615","title":"When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\\ell_1$-norm Lower Bounds","created":"2026-05-07","cited_by_count":2,"area":null},{"id":"2605.18528","title":"Tight Complexity of Muon under Heavy-Tailed Noise","created":"2026-05-18","cited_by_count":2,"area":null},{"id":"2606.14560","title":"Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success","created":"2026-06-12","cited_by_count":2,"area":null},{"id":"2606.19179","title":"Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods","created":"2026-06-17","cited_by_count":2,"area":null},{"id":"2607.01083","title":"Staleness-Learning Rate Scaling Laws for Asynchronous RLHF","created":"2026-07-01","cited_by_count":2,"area":"alignment"},{"id":"2311.02818","title":"Signal Processing Meets SGD: From Momentum to Filter","created":"2023-11-06","cited_by_count":1,"area":null},{"id":"2402.01621","title":"Stochastic Two Points Method for Deep Model Zeroth-order Optimization","created":"2024-02-02","cited_by_count":1,"area":null},{"id":"2403.18506","title":"Faster Convergence for Transformer Fine-tuning with Line Search Methods","created":"2024-03-27","cited_by_count":1,"area":null},{"id":"2403.18519","title":"Improving Line Search Methods for Large Scale Neural Network Training","created":"2024-03-27","cited_by_count":1,"area":null},{"id":"2406.17954","title":"Why Line Search when you can Plane Search? SO-Friendly Neural Networks allow Per-Iteration Optimization of Learning and Momentum Rates for Every Layer","created":"2024-06-25","cited_by_count":1,"area":null},{"id":"2407.02610","title":"Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point","created":"2024-07-02","cited_by_count":1,"area":null},{"id":"2407.20650","title":"No learning rates needed: Introducing SALSA -- Stable Armijo Line Search Adaptation","created":"2024-07-30","cited_by_count":1,"area":null},{"id":"2411.16462","title":"Lion Cub: Minimizing Communication Overhead in Distributed Lion","created":"2024-11-25","cited_by_count":1,"area":null},{"id":"2411.17866","title":"Distributed Sign Momentum with Local Steps for Training Transformers","created":"2024-11-26","cited_by_count":1,"area":null},{"id":"2412.02153","title":"Revisiting the Initial Steps in Adaptive Gradient Descent Optimization","created":"2024-12-03","cited_by_count":1,"area":null},{"id":"2503.03908","title":"On the Convergence of Adam-Type Algorithm for Bilevel Optimization under Unbounded Smoothness","created":"2025-03-05","cited_by_count":1,"area":null},{"id":"2507.06464","title":"SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam","created":"2025-07-09","cited_by_count":1,"area":null},{"id":"2601.18420","title":"Gradient Regularized Natural Gradients","created":"2026-01-26","cited_by_count":1,"area":null},{"id":"2601.21739","title":"Why $\\beta_1 = \\beta_2$ Is Dynamically Special in Adam","created":"2026-01-29","cited_by_count":1,"area":null},{"id":"2603.06120","title":"Dynamic Momentum Recalibration in Online Gradient Learning","created":"2026-03-06","cited_by_count":1,"area":null},{"id":"2605.04269","title":"Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization","created":"2026-05-05","cited_by_count":1,"area":null}],"next_cursor":"100"}