{"list":"other","total":105,"rows":[{"id":"1512.03385","title":"Deep Residual Learning for Image Recognition","created":"2015-12-10","cited_by_count":52096},{"id":"1502.03167","title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","created":"2015-02-11","cited_by_count":7005},{"id":"1607.06450","title":"Layer Normalization","created":"2016-07-21","cited_by_count":5669},{"id":"2104.09864","title":"RoFormer: Enhanced Transformer with Rotary Position Embedding","created":"2021-04-20","cited_by_count":4703},{"id":"1406.1078","title":"Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation","created":"2014-06-03","cited_by_count":4057},{"id":"2004.05150","title":"Longformer: The Long-Document Transformer","created":"2020-04-10","cited_by_count":3413},{"id":"2307.08691","title":"FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning","created":"2023-07-17","cited_by_count":2598},{"id":"1910.07467","title":"Root Mean Square Layer Normalization","created":"2019-10-16","cited_by_count":1480},{"id":"1804.04235","title":"Adafactor: Adaptive Learning Rates with Sublinear Memory Cost","created":"2018-04-11","cited_by_count":905},{"id":"1904.09237","title":"On the Convergence of Adam and Beyond","created":"2019-04-19","cited_by_count":841},{"id":"1503.05671","title":"Optimizing Neural Networks with Kronecker-factored Approximate Curvature","created":"2015-03-19","cited_by_count":796},{"id":"1309.5549","title":"Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming","created":"2013-09-22","cited_by_count":724},{"id":"1904.00962","title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","created":"2019-04-01","cited_by_count":674},{"id":"1802.04434","title":"signSGD: Compressed Optimisation for Non-Convex Problems","created":"2018-02-13","cited_by_count":571},{"id":"1708.03888","title":"Large Batch Training of Convolutional Networks","created":"2017-08-13","cited_by_count":535},{"id":"2403.03507","title":"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection","created":"2024-03-06","cited_by_count":463},{"id":"2302.06675","title":"Symbolic Discovery of Optimization Algorithms","created":"2023-02-13","cited_by_count":434},{"id":"2502.16982","title":"Muon is Scalable for LLM Training","created":"2025-02-24","cited_by_count":428},{"id":"1802.09568","title":"Shampoo: Preconditioned Stochastic Tensor Optimization","created":"2018-02-26","cited_by_count":426},{"id":"2406.11794","title":"DataComp-LM: In search of the next generation of training sets for language models","created":"2024-06-17","cited_by_count":398},{"id":"1901.09847","title":"Error Feedback Fixes SignSGD and other Gradient Compression Schemes","created":"2019-01-28","cited_by_count":310},{"id":"1912.02365","title":"Lower Bounds for Non-Convex Stochastic Optimization","created":"2019-12-05","cited_by_count":307},{"id":"2203.03466","title":"Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer","created":"2022-03-07","cited_by_count":288},{"id":"2010.04245","title":"Query-Key Normalization for Transformers","created":"2020-10-08","cited_by_count":280},{"id":"1901.06053","title":"A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks","created":"2019-01-18","cited_by_count":214},{"id":"2409.20325","title":"Old Optimizer, New Norm: An Anthology","created":"2024-09-30","cited_by_count":193},{"id":"2502.07529","title":"Training Deep Learning Models with Norm-Constrained LMOs","created":"2025-02-11","cited_by_count":192},{"id":"2208.06677","title":"Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models","created":"2022-08-13","cited_by_count":174},{"id":"2405.15682","title":"The Road Less Scheduled","created":"2024-05-24","cited_by_count":163},{"id":"1602.01407","title":"A Kronecker-factored approximate Fisher matrix for convolution layers","created":"2016-02-03","cited_by_count":141},{"id":"1705.07774","title":"Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients","created":"2017-05-22","cited_by_count":139},{"id":"2002.03305","title":"Momentum Improves Normalized SGD","created":"2020-02-09","cited_by_count":139},{"id":"2006.04740","title":"The Heavy-Tail Phenomenon in SGD","created":"2020-06-08","cited_by_count":124},{"id":"2208.09632","title":"Adam Can Converge Without Any Modification On Update Rules","created":"2022-08-20","cited_by_count":114},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114},{"id":"2505.16932","title":"The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm","created":"2025-05-22","cited_by_count":110},{"id":"2310.17813","title":"A Spectral Condition for Feature Learning","created":"2023-10-26","cited_by_count":109},{"id":"2304.13972","title":"Convergence of Adam Under Relaxed Assumptions","created":"2023-04-27","cited_by_count":99},{"id":"2503.12645","title":"Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization","created":"2025-03-16","cited_by_count":92},{"id":"2506.15054","title":"Muon Optimizes Under Spectral Norm Constraints","created":"2025-06-18","cited_by_count":86},{"id":"2510.05491","title":"NorMuon: Making Muon more efficient and scalable","created":"2025-10-07","cited_by_count":84},{"id":"2502.02900","title":"A Note on the Convergence of Muon","created":"2025-02-05","cited_by_count":83},{"id":"2106.14343","title":"High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails","created":"2021-06-28","cited_by_count":80},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80},{"id":"2505.01618","title":"Don't be lazy: CompleteP enables compute-efficient deep transformers","created":"2025-05-02","cited_by_count":78},{"id":"2410.21265","title":"Modular Duality in Deep Learning","created":"2024-10-28","cited_by_count":77},{"id":"2310.04415","title":"Why Do We Need Weight Decay in Modern Deep Learning?","created":"2023-10-06","cited_by_count":72},{"id":"2505.13416","title":"Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)","created":"2025-05-19","cited_by_count":72},{"id":"2006.08877","title":"Practical Quasi-Newton Methods for Training Deep Neural Networks","created":"2020-06-16","cited_by_count":68},{"id":"2405.14813","title":"Scalable Optimization in the Modular Norm","created":"2024-05-23","cited_by_count":68},{"id":"2504.13161","title":"Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training","created":"2025-04-17","cited_by_count":66},{"id":"2411.10438","title":"MARS: Unleashing the Power of Variance Reduction for Training Large Models","created":"2024-11-15","cited_by_count":64},{"id":"2302.00999","title":"High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variance","created":"2023-02-02","cited_by_count":61},{"id":"2406.17748","title":"A New Perspective on Shampoo's Preconditioner","created":"2024-06-25","cited_by_count":59},{"id":"2404.04454","title":"Implicit Bias of AdamW: $\\ell_\\infty$ Norm Constrained Optimization","created":"2024-04-05","cited_by_count":58},{"id":"2305.18471","title":"Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptions","created":"2023-05-29","cited_by_count":57},{"id":"2503.20762","title":"ASGO: Adaptive Structured Gradient Optimization","created":"2025-03-26","cited_by_count":57},{"id":"2505.21799","title":"PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective","created":"2025-05-27","cited_by_count":52},{"id":"2310.05898","title":"Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts","created":"2023-10-09","cited_by_count":51},{"id":"1905.12938","title":"Stochastic Sign Descent Methods: New Algorithms and Better Theory","created":"2019-05-30","cited_by_count":49},{"id":"2410.13849","title":"From Gradient Clipping to Normalization for Heavy Tailed SGD","created":"2024-10-17","cited_by_count":49},{"id":"2412.19529","title":"Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clipping","created":"2024-12-27","cited_by_count":42},{"id":"2502.04664","title":"Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data","created":"2025-02-07","cited_by_count":41},{"id":"2506.04192","title":"Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noise","created":"2025-06-04","cited_by_count":41},{"id":"2511.00674","title":"Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?","created":"2025-11-01","cited_by_count":37},{"id":"2601.19156","title":"Convergence of Muon with Newton-Schulz","created":"2026-01-27","cited_by_count":36},{"id":"2601.13474","title":"Preconditioning Benefits of Spectral Orthogonalization in Muon","created":"2026-01-20","cited_by_count":30},{"id":"2210.14355","title":"Parameter-free Regret in High Probability with Heavy Tails","created":"2022-10-25","cited_by_count":28},{"id":"2507.01598","title":"Convergence Bound and Critical Batch Size of Muon Optimizer","created":"2025-07-02","cited_by_count":28},{"id":"2302.06763","title":"Breaking the Lower Bound with (Little) Structure: Acceleration in Non-Convex Stochastic Optimization with Heavy-Tailed Noise","created":"2023-02-14","cited_by_count":27},{"id":"2410.17897","title":"Value Residual Learning","created":"2024-10-23","cited_by_count":27},{"id":"2004.13465","title":"Nearly Optimal Regret for Stochastic Linear Bandits with Heavy-Tailed Payoffs","created":"2020-04-28","cited_by_count":25},{"id":"2106.02925","title":"Tensor Normal Training for Deep Learning Models","created":"2021-06-05","cited_by_count":24},{"id":"2202.11632","title":"Mirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise Variance","created":"2022-02-23","cited_by_count":24},{"id":"2310.01860","title":"High-Probability Convergence for Composite and Distributed Stochastic Minimization and Variational Inequalities with Heavy-Tailed Noise","created":"2023-10-03","cited_by_count":24},{"id":"2505.19227","title":"Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law","created":"2025-05-25","cited_by_count":24},{"id":"2409.15156","title":"Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling","created":"2024-09-23","cited_by_count":23},{"id":"2406.15244","title":"AdaGrad under Anisotropic Smoothness","created":"2024-06-21","cited_by_count":22},{"id":"2509.14562","title":"LiMuon: Light and Fast Muon Optimizer for Large Models","created":"2025-09-18","cited_by_count":22},{"id":"2512.05620","title":"Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales","created":"2025-12-05","cited_by_count":21},{"id":"2406.04443","title":"Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed","created":"2024-06-06","cited_by_count":18},{"id":"2410.16561","title":"Revisiting Gradient Normalization and Clipping for Nonconvex SGD under Heavy-Tailed Noise: Necessity, Sufficiency, and Acceleration","created":"2024-10-21","cited_by_count":18},{"id":"2509.11983","title":"Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training","created":"2025-09-15","cited_by_count":18},{"id":"2604.01472","title":"The Newton-Muon Optimizer","created":"2026-04-01","cited_by_count":18},{"id":"2508.07473","title":"Online Convex Optimization with Heavy Tails: Old Algorithms, New Regrets, and Applications","created":"2025-08-10","cited_by_count":17},{"id":"2404.00498","title":"94% on CIFAR-10 in 3.29 Seconds on a Single GPU","created":"2024-03-30","cited_by_count":16},{"id":"2406.04592","title":"Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimization","created":"2024-06-07","cited_by_count":16},{"id":"2602.07425","title":"Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise","created":"2026-02-07","cited_by_count":15},{"id":"2602.08232","title":"Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimization","created":"2026-02-09","cited_by_count":14},{"id":"2602.17080","title":"Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum","created":"2026-02-19","cited_by_count":14},{"id":"2605.11181","title":"Muon is Not That Special: Random or Inverted Spectra Work Just as Well","created":"2026-05-11","cited_by_count":14},{"id":"2512.16928","title":"Dion2: A Simple Method to Shrink Matrix in Muon","created":"2025-12-01","cited_by_count":13},{"id":"2602.11948","title":"Insights on Muon from Simple Quadratics","created":"2026-02-12","cited_by_count":13},{"id":"2601.23000","title":"Mano: Restriking Manifold Optimization for LLM Training","created":"2026-01-30","cited_by_count":12},{"id":"2310.18701","title":"Efficient Algorithms for Generalized Linear Bandits with Heavy-tailed Rewards","created":"2023-10-28","cited_by_count":11},{"id":"2601.21750","title":"FISMO: Fisher-Structured Momentum-Orthogonalized Optimizer","created":"2026-01-29","cited_by_count":11},{"id":"2602.16340","title":"The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks","created":"2026-02-18","cited_by_count":11},{"id":"2603.00742","title":"To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters","created":"2026-02-28","cited_by_count":10},{"id":"2605.08980","title":"Muon Does Not Converge on Convex Lipschitz Functions","created":"2026-05-09","cited_by_count":9},{"id":"2605.06884","title":"Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition","created":"2026-05-07","cited_by_count":8}],"next_cursor":"100"}