{"total":117,"rows":[{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80,"area":null},{"id":"2109.14119","title":"Stochastic Training is Not Necessary for Generalization","created":"2021-09-29","cited_by_count":63,"area":null},{"id":"2211.09760","title":"VeLO: Training Versatile Learned Optimizers by Scaling Up","created":"2022-11-17","cited_by_count":57,"area":null},{"id":"2105.02716","title":"Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks","created":"2021-05-06","cited_by_count":50,"area":null},{"id":"2107.00940","title":"Inverse-Dirichlet Weighting Enables Reliable Training of Physics Informed Neural Networks","created":"2021-07-02","cited_by_count":43,"area":null},{"id":"2307.06440","title":"No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models","created":"2023-07-12","cited_by_count":41,"area":null},{"id":"2407.07972","title":"Deconstructing What Makes a Good Optimizer for Language Models","created":"2024-07-10","cited_by_count":41,"area":null},{"id":"2210.06640","title":"Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities","created":"2022-10-13","cited_by_count":38,"area":null},{"id":"2502.15938","title":"Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs","created":"2025-02-21","cited_by_count":35,"area":null},{"id":"2203.11860","title":"Practical tradeoffs between memory, compute, and performance in learned optimizers","created":"2022-03-22","cited_by_count":32,"area":null},{"id":"2502.15015","title":"Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition","created":"2025-02-20","cited_by_count":32,"area":null},{"id":"2209.11208","title":"A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases","created":"2022-09-22","cited_by_count":30,"area":null},{"id":"2101.11075","title":"Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization","created":"2021-01-26","cited_by_count":28,"area":null},{"id":"2204.06815","title":"deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks","created":"2022-04-14","cited_by_count":28,"area":null},{"id":"2206.13424","title":"Benchopt: Reproducible, efficient and collaborative optimization benchmarks","created":"2022-06-27","cited_by_count":27,"area":null},{"id":"2303.08011","title":"Model scale versus domain knowledge in statistical forecasting of chaotic systems","created":"2023-03-13","cited_by_count":27,"area":null},{"id":"2106.02925","title":"Tensor Normal Training for Deep Learning Models","created":"2021-06-05","cited_by_count":24,"area":null},{"id":"2305.07583","title":"MoMo: Momentum Models for Adaptive Learning Rates","created":"2023-05-12","cited_by_count":24,"area":null},{"id":"2106.13731","title":"Ranger21: a synergistic deep learning optimizer","created":"2021-06-25","cited_by_count":22,"area":null},{"id":"2102.06356","title":"A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes","created":"2021-02-12","cited_by_count":20,"area":null},{"id":"2310.17653","title":"Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model","created":"2023-10-26","cited_by_count":19,"area":null},{"id":"2203.06498","title":"The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning","created":"2022-03-12","cited_by_count":17,"area":null},{"id":"2106.02624","title":"ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure","created":"2021-06-04","cited_by_count":16,"area":null},{"id":"2510.25000","title":"What Really Matters in Matrix-Whitening Optimizers?","created":"2025-10-28","cited_by_count":16,"area":null},{"id":"2304.05187","title":"Automatic Gradient Descent: Deep Learning without Hyperparameters","created":"2023-04-11","cited_by_count":14,"area":null},{"id":"2310.11428","title":"Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression","created":"2023-10-17","cited_by_count":14,"area":null},{"id":"2507.18807","title":"Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator","created":"2025-07-24","cited_by_count":14,"area":null},{"id":"2206.00395","title":"Optimization with Access to Auxiliary Information","created":"2022-06-01","cited_by_count":13,"area":null},{"id":"2305.02665","title":"Learning Language-Specific Layers for Multilingual Machine Translation","created":"2023-05-04","cited_by_count":13,"area":null},{"id":"2308.07832","title":"REFORMS: Reporting Standards for Machine Learning Based Science","created":"2023-08-15","cited_by_count":13,"area":null},{"id":"2201.06714","title":"AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimization","created":"2022-01-18","cited_by_count":11,"area":null},{"id":"2206.05985","title":"Modeling the Machine Learning Multiverse","created":"2022-06-13","cited_by_count":11,"area":null},{"id":"2104.04657","title":"Meta-Learning Bidirectional Update Rules","created":"2021-04-10","cited_by_count":10,"area":null},{"id":"2203.14177","title":"Benchmarking Deep AUROC Optimization: Loss Functions and Algorithmic Choices","created":"2022-03-27","cited_by_count":10,"area":null},{"id":"2407.04358","title":"An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes","created":"2024-07-05","cited_by_count":10,"area":null},{"id":"2407.13954","title":"Neural topology optimization: the good, the bad, and the ugly","created":"2024-07-19","cited_by_count":10,"area":null},{"id":"2602.04998","title":"Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning","created":"2026-02-04","cited_by_count":10,"area":null},{"id":"2202.04124","title":"A Mini-Block Fisher Method for Deep Neural Networks","created":"2022-02-08","cited_by_count":8,"area":null},{"id":"2203.01400","title":"Adaptive Gradient Methods with Local Guarantees","created":"2022-03-02","cited_by_count":8,"area":null},{"id":"2302.09726","title":"Nystrom Method for Accurate and Scalable Implicit Differentiation","created":"2023-02-20","cited_by_count":8,"area":null},{"id":"2407.20209","title":"Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learning","created":"2024-07-29","cited_by_count":8,"area":null},{"id":"2602.09314","title":"Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory","created":"2026-02-10","cited_by_count":8,"area":null},{"id":"2107.01832","title":"Provable Convergence of Nesterov's Accelerated Gradient Method for Over-Parameterized Neural Networks","created":"2021-07-05","cited_by_count":7,"area":null},{"id":"2201.06118","title":"DeepCreativity: Measuring Creativity with Deep Learning Techniques","created":"2022-01-16","cited_by_count":7,"area":null},{"id":"2106.11514","title":"Rethinking Adam: A Twofold Exponential Moving Average Approach","created":"2021-06-22","cited_by_count":6,"area":null},{"id":"2111.04260","title":"Personalized Benchmarking with the Ludwig Benchmarking Toolkit","created":"2021-11-08","cited_by_count":6,"area":null},{"id":"2310.00137","title":"On the Disconnect Between Theory and Practice of Neural Networks: Limits of the NTK Perspective","created":"2023-09-29","cited_by_count":6,"area":null},{"id":"2401.08893","title":"MADA: Meta-Adaptive Optimizers through hyper-gradient Descent","created":"2024-01-17","cited_by_count":6,"area":null},{"id":"2602.07712","title":"Towards Robust Scaling Laws for Optimizers","created":"2026-02-07","cited_by_count":6,"area":null},{"id":"2111.04004","title":"Exponential escape efficiency of SGD from sharp minima in non-stationary regime","created":"2021-11-07","cited_by_count":5,"area":null},{"id":"2205.10577","title":"Non-Autoregressive Neural Machine Translation: A Call for Clarity","created":"2022-05-21","cited_by_count":5,"area":null},{"id":"2206.00832","title":"Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates","created":"2022-06-02","cited_by_count":5,"area":null},{"id":"2504.01898","title":"Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation","created":"2025-04-02","cited_by_count":5,"area":null},{"id":"2603.29791","title":"Reasoning-Driven Synthetic Data Generation and Evaluation","created":"2026-03-31","cited_by_count":5,"area":null},{"id":"2108.09134","title":"Accelerating Federated Learning with a Global Biased Optimiser","created":"2021-08-20","cited_by_count":4,"area":null},{"id":"2204.08306","title":"A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networks","created":"2022-04-18","cited_by_count":4,"area":null},{"id":"2212.07624","title":"Neuroevolution of Physics-Informed Neural Nets: Benchmark Problems and Comparative Results","created":"2022-12-15","cited_by_count":4,"area":null},{"id":"2308.04457","title":"A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems","created":"2023-08-06","cited_by_count":4,"area":null},{"id":"2311.02394","title":"NeuroEvoBench: Benchmarking Evolutionary Optimizers for Deep Learning Applications","created":"2023-11-04","cited_by_count":4,"area":null},{"id":"2501.12670","title":"Celo: Training Versatile Learned Optimizers on a Compute Diet","created":"2025-01-22","cited_by_count":4,"area":null},{"id":"2602.01642","title":"The Effect of Mini-Batch Noise on the Implicit Bias of Adam","created":"2026-02-02","cited_by_count":4,"area":null},{"id":"2211.08583","title":"Empirical Study on Optimizer Selection for Out-of-Distribution Generalization","created":"2022-11-15","cited_by_count":3,"area":null},{"id":"2211.15596","title":"A survey of deep learning optimizers -- first and second order methods","created":"2022-11-28","cited_by_count":3,"area":null},{"id":"2303.08169","title":"Allegro-Legato: Scalable, Fast, and Robust Neural-Network Quantum Molecular Dynamics via Sharpness-Aware Minimization","created":"2023-03-14","cited_by_count":3,"area":null},{"id":"2310.10006","title":"Soft ascent-descent as a stable and flexible alternative to flooding","created":"2023-10-16","cited_by_count":3,"area":null},{"id":"2408.09310","title":"Narrowing the Focus: Learned Optimizers for Pretrained Models","created":"2024-08-17","cited_by_count":3,"area":null},{"id":"2203.01603","title":"AdaFamily: A family of Adam-like adaptive gradient methods","created":"2022-03-03","cited_by_count":2,"area":null},{"id":"2206.11533","title":"Stochastic Langevin Differential Inclusions with Applications to Machine Learning","created":"2022-06-23","cited_by_count":2,"area":null},{"id":"2306.05300","title":"Anti-Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances in Flat Directions","created":"2023-06-08","cited_by_count":2,"area":null},{"id":"2310.18191","title":"Is Scaling Learned Optimizers Worth It? Evaluating The Value of VeLO's 4000 TPU Months","created":"2023-10-27","cited_by_count":2,"area":null},{"id":"2404.00390","title":"Learning truly monotone operators with applications to nonlinear inverse problems","created":"2024-03-30","cited_by_count":2,"area":null},{"id":"2410.14325","title":"Debiasing Mini-Batch Quadratics for Applications in Deep Learning","created":"2024-10-18","cited_by_count":2,"area":null},{"id":"2505.24005","title":"How far away are truly hyperparameter-free learning algorithms?","created":"2025-05-29","cited_by_count":2,"area":null},{"id":"2506.06558","title":"Rapid training of Hamiltonian graph networks using random features","created":"2025-06-06","cited_by_count":2,"area":null},{"id":"2511.19716","title":"Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability","created":"2025-11-24","cited_by_count":2,"area":null},{"id":"2512.06370","title":"Greedy Alignment Principle for Optimizer Selection","created":"2025-12-06","cited_by_count":2,"area":null},{"id":"2102.08716","title":"Training Aware Sigmoidal Optimizer","created":"2021-02-17","cited_by_count":1,"area":null},{"id":"2104.01742","title":"Explainability-aided Domain Generalization for Image Classification","created":"2021-04-05","cited_by_count":1,"area":null},{"id":"2108.11713","title":"The Number of Steps Needed for Nonconvex Optimization of a Deep Learning Optimizer is a Rational Function of Batch Size","created":"2021-08-26","cited_by_count":1,"area":null},{"id":"2203.06474","title":"Optimizer Amalgamation","created":"2022-03-12","cited_by_count":1,"area":null},{"id":"2305.01377","title":"Random Function Descent","created":"2023-05-02","cited_by_count":1,"area":null},{"id":"2309.12095","title":"Bayesian sparsification for deep neural networks with Bayesian model reduction","created":"2023-09-21","cited_by_count":1,"area":null},{"id":"2311.16956","title":"Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent","created":"2023-11-28","cited_by_count":1,"area":null},{"id":"2311.18598","title":"Generalisable Agents for Neural Network Optimisation","created":"2023-11-30","cited_by_count":1,"area":null},{"id":"2402.06948","title":"Should I try multiple optimizers when fine-tuning pre-trained Transformers for NLP tasks? Should I tune their hyperparameters?","created":"2024-02-10","cited_by_count":1,"area":null},{"id":"2403.18519","title":"Improving Line Search Methods for Large Scale Neural Network Training","created":"2024-03-27","cited_by_count":1,"area":null},{"id":"2404.17651","title":"Hard ASH: Sparsity and the right optimizer make a continual learner","created":"2024-04-26","cited_by_count":1,"area":null},{"id":"2407.20650","title":"No learning rates needed: Introducing SALSA -- Stable Armijo Line Search Adaptation","created":"2024-07-30","cited_by_count":1,"area":null},{"id":"2409.11995","title":"Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapes","created":"2024-09-18","cited_by_count":1,"area":null},{"id":"2412.17082","title":"MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes","created":"2024-12-22","cited_by_count":1,"area":null},{"id":"2508.18912","title":"HOTSPOT-YOLO: A Lightweight Deep Learning Attention-Driven Model for Detecting Thermal Anomalies in Drone-Based Solar Photovoltaic Inspections","created":"2025-08-26","cited_by_count":1,"area":null},{"id":"2601.07830","title":"Optimal Learning Rate Schedule for Balancing Effort and Performance","created":"2026-01-12","cited_by_count":1,"area":null},{"id":"2606.14259","title":"Beyond a Single Explanation of the Adam--SGD Gap","created":"2026-06-12","cited_by_count":1,"area":null},{"id":"2105.14625","title":"Surrogate Model Based Hyperparameter Tuning for Deep Learning with SPOT","created":"2021-05-30","cited_by_count":0,"area":null},{"id":"2112.07163","title":"Minimization of Stochastic First-order Oracle Complexity of Adaptive Methods for Nonconvex Optimization","created":"2021-12-14","cited_by_count":0,"area":null},{"id":"2207.12013","title":"Effective and Interpretable Information Aggregation with Capacity Networks","created":"2022-07-25","cited_by_count":0,"area":null},{"id":"2207.12505","title":"On the benefits of non-linear weight updates","created":"2022-07-25","cited_by_count":0,"area":null},{"id":"2208.02017","title":"Neural Nets with a Newton Conjugate Gradient Method on Multiple GPUs","created":"2022-08-03","cited_by_count":0,"area":null},{"id":"2208.03941","title":"Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks","created":"2022-08-08","cited_by_count":0,"area":null},{"id":"2212.04875","title":"Expeditious Saliency-guided Mix-up through Random Gradient Thresholding","created":"2022-12-09","cited_by_count":0,"area":null}],"next_cursor":"100"}