{"list":"other","total":59,"rows":[{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"1503.05671","title":"Optimizing Neural Networks with Kronecker-factored Approximate Curvature","created":"2015-03-19","cited_by_count":796},{"id":"2301.00808","title":"ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders","created":"2023-01-02","cited_by_count":712},{"id":"1802.04434","title":"signSGD: Compressed Optimisation for Non-Convex Problems","created":"2018-02-13","cited_by_count":571},{"id":"1308.0850","title":"Generating Sequences With Recurrent Neural Networks","created":"2013-08-04","cited_by_count":554},{"id":"1412.1193","title":"New insights and perspectives on the natural gradient method","created":"2014-12-03","cited_by_count":443},{"id":"2502.16982","title":"Muon is Scalable for LLM Training","created":"2025-02-24","cited_by_count":428},{"id":"2010.07468","title":"AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients","created":"2020-10-15","cited_by_count":283},{"id":"1905.12558","title":"Limitations of the Empirical Fisher Approximation for Natural Gradient Descent","created":"2019-05-29","cited_by_count":217},{"id":"2409.20325","title":"Old Optimizer, New Norm: An Anthology","created":"2024-09-30","cited_by_count":193},{"id":"2409.11321","title":"SOAP: Improving and Stabilizing Shampoo using Adam","created":"2024-09-17","cited_by_count":192},{"id":"2502.07529","title":"Training Deep Learning Models with Norm-Constrained LMOs","created":"2025-02-11","cited_by_count":192},{"id":"1806.03884","title":"Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis","created":"2018-06-11","cited_by_count":146},{"id":"1602.01407","title":"A Kronecker-factored approximate Fisher matrix for convolution layers","created":"2016-02-03","cited_by_count":141},{"id":"1705.07774","title":"Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients","created":"2017-05-22","cited_by_count":139},{"id":"2007.01547","title":"Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers","created":"2020-07-03","cited_by_count":119},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114},{"id":"2505.16932","title":"The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm","created":"2025-05-22","cited_by_count":110},{"id":"2310.17813","title":"A Spectral Condition for Feature Learning","created":"2023-10-26","cited_by_count":109},{"id":"2509.02046","title":"Fantastic Pretraining Optimizers and Where to Find Them","created":"2025-09-02","cited_by_count":100},{"id":"2205.01580","title":"Better plain ViT baselines for ImageNet-1k","created":"2022-05-03","cited_by_count":98},{"id":"1512.00809","title":"Optimal whitening and decorrelation","created":"2015-12-02","cited_by_count":97},{"id":"2410.06511","title":"TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training","created":"2024-10-09","cited_by_count":85},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80},{"id":"2309.06497","title":"A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale","created":"2023-09-12","cited_by_count":77},{"id":"2410.21265","title":"Modular Duality in Deep Learning","created":"2024-10-28","cited_by_count":77},{"id":"2405.14813","title":"Scalable Optimization in the Modular Norm","created":"2024-05-23","cited_by_count":68},{"id":"2509.01440","title":"Benchmarking Optimizers for Large Language Model Pretraining","created":"2025-09-01","cited_by_count":68},{"id":"2410.24206","title":"Understanding Optimization in Deep Learning with Central Flows","created":"2024-10-31","cited_by_count":64},{"id":"2411.10438","title":"MARS: Unleashing the Power of Variance Reduction for Training Large Models","created":"2024-11-15","cited_by_count":64},{"id":"2406.17748","title":"A New Perspective on Shampoo's Preconditioner","created":"2024-06-25","cited_by_count":59},{"id":"2503.20762","title":"ASGO: Adaptive Structured Gradient Optimization","created":"2025-03-26","cited_by_count":57},{"id":"2409.03137","title":"The AdEMAMix Optimizer: Better, Faster, Older","created":"2024-09-05","cited_by_count":54},{"id":"1906.02914","title":"Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations","created":"2019-06-07","cited_by_count":51},{"id":"2311.00636","title":"Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures","created":"2023-11-01","cited_by_count":49},{"id":"2509.26030","title":"Muon Outperforms Adam in Tail-End Associative Memory Learning","created":"2025-09-30","cited_by_count":46},{"id":"2002.11803","title":"Disentangling Adaptive Gradient Methods from Learning Rates","created":"2020-02-26","cited_by_count":42},{"id":"2002.08056","title":"The Geometry of Sign Gradient Descent","created":"2020-02-19","cited_by_count":41},{"id":"2507.07101","title":"Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful","created":"2025-07-09","cited_by_count":41},{"id":"2505.21829","title":"In Search of Adam's Secret Sauce","created":"2025-05-27","cited_by_count":40},{"id":"2511.00674","title":"Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?","created":"2025-11-01","cited_by_count":37},{"id":"2503.10537","title":"Structured Preconditioners in Adaptive Optimization: A Unified Analysis","created":"2025-03-13","cited_by_count":33},{"id":"2502.15015","title":"Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition","created":"2025-02-20","cited_by_count":32},{"id":"2512.04299","title":"When do spectral gradient updates help in deep learning?","created":"2025-12-03","cited_by_count":31},{"id":"2201.12250","title":"Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization","created":"2022-01-28","cited_by_count":30},{"id":"2505.19227","title":"Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Law","created":"2025-05-25","cited_by_count":24},{"id":"2410.08198","title":"Adam Exploits $\\ell_\\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity","created":"2024-10-10","cited_by_count":23},{"id":"2506.03595","title":"Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner","created":"2025-06-04","cited_by_count":23},{"id":"2002.04839","title":"LaProp: Separating Momentum and Adaptivity in Adam","created":"2020-02-12","cited_by_count":20},{"id":"1304.2338","title":"An Almost-Linear-Time Algorithm for Approximate Max Flow in Undirected Graphs, and its Multicommodity Generalizations","created":"2013-04-08","cited_by_count":19},{"id":"2507.12224","title":"Optimizers Qualitatively Alter Solutions And We Should Leverage This","created":"2025-07-16","cited_by_count":17},{"id":"2510.25000","title":"What Really Matters in Matrix-Whitening Optimizers?","created":"2025-10-28","cited_by_count":16},{"id":"2410.19964","title":"Understanding Adam Requires Better Rotation Dependent Assumptions","created":"2024-10-25","cited_by_count":11},{"id":"2405.19279","title":"Understanding and Minimising Outlier Features in Neural Network Training","created":"2024-05-29","cited_by_count":10},{"id":"1807.07540","title":"Bayesian filtering unifies adaptive and non-adaptive neural network optimization methods","created":"2018-07-19","cited_by_count":9},{"id":"2502.02132","title":"How Memory in Optimization Algorithms Implicitly Modifies the Loss","created":"2025-02-04","cited_by_count":6},{"id":"2509.23500","title":"Beyond Outliers: A Study of Optimizers Under Quantization","created":"2025-09-27","cited_by_count":6},{"id":"2511.20584","title":"A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descent","created":"2025-11-25","cited_by_count":5},{"id":"2507.08963","title":"Stochastic Approximation with Block Coordinate Optimal Stepsizes","created":"2025-07-11","cited_by_count":3}],"next_cursor":null}