{"total":124,"rows":[{"id":"2406.16793","title":"Adam-mini: Use Fewer Learning Rates To Gain More","created":"2024-06-24","cited_by_count":137,"area":null},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80,"area":null},{"id":"2408.16673","title":"Preserving Diversity in Supervised Fine-Tuning of Large Language Models","created":"2024-08-29","cited_by_count":70,"area":null},{"id":"2503.20762","title":"ASGO: Adaptive Structured Gradient Optimization","created":"2025-03-26","cited_by_count":57,"area":null},{"id":"2412.05270","title":"APOLLO: SGD-like Memory, AdamW-level Performance","created":"2024-12-06","cited_by_count":54,"area":null},{"id":"2509.26030","title":"Muon Outperforms Adam in Tail-End Associative Memory Learning","created":"2025-09-30","cited_by_count":46,"area":null},{"id":"2407.07972","title":"Deconstructing What Makes a Good Optimizer for Language Models","created":"2024-07-10","cited_by_count":41,"area":null},{"id":"2502.04664","title":"Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data","created":"2025-02-07","cited_by_count":41,"area":null},{"id":"2507.07101","title":"Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful","created":"2025-07-09","cited_by_count":41,"area":null},{"id":"2505.21829","title":"In Search of Adam's Secret Sauce","created":"2025-05-27","cited_by_count":40,"area":null},{"id":"2511.00674","title":"Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?","created":"2025-11-01","cited_by_count":37,"area":null},{"id":"2405.16002","title":"Does SGD really happen in tiny subspaces?","created":"2024-05-25","cited_by_count":36,"area":null},{"id":"2502.02732","title":"Peri-LN: Revisiting Normalization Layer in the Transformer Architecture","created":"2025-02-04","cited_by_count":34,"area":null},{"id":"2502.17410","title":"COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs","created":"2025-02-24","cited_by_count":30,"area":null},{"id":"2502.19002","title":"The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training","created":"2025-02-26","cited_by_count":30,"area":null},{"id":"2601.13474","title":"Preconditioning Benefits of Spectral Orthogonalization in Muon","created":"2026-01-20","cited_by_count":30,"area":null},{"id":"2505.17863","title":"The emergence of sparse attention: impact of data distribution and benefits of repetition","created":"2025-05-23","cited_by_count":25,"area":null},{"id":"2405.18199","title":"Adam with model exponential moving average is effective for nonconvex optimization","created":"2024-05-28","cited_by_count":23,"area":null},{"id":"2410.08198","title":"Adam Exploits $\\ell_\\infty$-geometry of Loss Landscape via Coordinate-wise Adaptivity","created":"2024-10-10","cited_by_count":23,"area":null},{"id":"2406.15244","title":"AdaGrad under Anisotropic Smoothness","created":"2024-06-21","cited_by_count":22,"area":null},{"id":"2412.13148","title":"SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training","created":"2024-12-17","cited_by_count":20,"area":null},{"id":"2410.02486","title":"Encryption-Friendly LLM Architecture","created":"2024-10-03","cited_by_count":17,"area":null},{"id":"2504.07097","title":"Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning","created":"2025-04-09","cited_by_count":15,"area":null},{"id":"2506.12543","title":"Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling","created":"2025-06-14","cited_by_count":15,"area":null},{"id":"2510.27486","title":"FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models","created":"2025-10-31","cited_by_count":15,"area":null},{"id":"2603.20527","title":"RMNP: Row-Momentum Normalized Preconditioning for Scalable Matrix-Based Optimization","created":"2026-03-20","cited_by_count":15,"area":null},{"id":"2405.16397","title":"AdaFisher: Adaptive Second Order Optimization via Fisher Information","created":"2024-05-26","cited_by_count":14,"area":null},{"id":"2407.20999","title":"MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning","created":"2024-07-30","cited_by_count":14,"area":null},{"id":"2607.21716","title":"A Defense of the Quadratic Model","created":"2026-07-23","cited_by_count":14,"area":null},{"id":"2505.02809","title":"Towards Quantifying the Hessian Structure of Neural Networks","created":"2025-05-05","cited_by_count":13,"area":null},{"id":"2505.17646","title":"Unveiling the Basin-Like Loss Landscape in Large Language Models","created":"2025-05-23","cited_by_count":13,"area":null},{"id":"2505.24022","title":"The Rich and the Simple: On the Implicit Bias of Adam and SGD","created":"2025-05-29","cited_by_count":13,"area":null},{"id":"2410.06441","title":"Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models","created":"2024-10-09","cited_by_count":11,"area":null},{"id":"2410.19964","title":"Understanding Adam Requires Better Rotation Dependent Assumptions","created":"2024-10-25","cited_by_count":11,"area":null},{"id":"2411.07724","title":"Convergence Rate Analysis of LION","created":"2024-11-12","cited_by_count":11,"area":null},{"id":"2602.04998","title":"Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning","created":"2026-02-04","cited_by_count":10,"area":null},{"id":"2410.10986","title":"What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis","created":"2024-10-14","cited_by_count":9,"area":null},{"id":"2506.01901","title":"Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods","created":"2025-06-02","cited_by_count":9,"area":null},{"id":"2412.11768","title":"No More Adam: Learning Rate Scaling at Initialization is All You Need","created":"2024-12-16","cited_by_count":8,"area":null},{"id":"2502.06244","title":"PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts","created":"2025-02-10","cited_by_count":8,"area":null},{"id":"2505.24275","title":"GradPower: Powering Gradients for Faster Language Model Pre-Training","created":"2025-05-30","cited_by_count":8,"area":null},{"id":"2602.22681","title":"Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement","created":"2026-02-26","cited_by_count":8,"area":null},{"id":"2410.05090","title":"HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimation","created":"2024-10-07","cited_by_count":7,"area":null},{"id":"2411.12135","title":"Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effects","created":"2024-11-19","cited_by_count":7,"area":null},{"id":"2510.03866","title":"On Provable Benefits of Muon in Federated Learning","created":"2025-10-04","cited_by_count":7,"area":null},{"id":"2602.07729","title":"Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs","created":"2026-02-07","cited_by_count":7,"area":null},{"id":"2602.15322","title":"On Surprising Effectiveness of Masking Updates in Adaptive Optimizers","created":"2026-02-17","cited_by_count":7,"area":null},{"id":"2505.15624","title":"Mechanistic Insights into Grokking from the Embedding Layer","created":"2025-05-21","cited_by_count":6,"area":null},{"id":"2511.21622","title":"On the Origin of Algorithmic Progress in AI","created":"2025-11-26","cited_by_count":6,"area":null},{"id":"2602.07712","title":"Towards Robust Scaling Laws for Optimizers","created":"2026-02-07","cited_by_count":6,"area":null},{"id":"2602.21545","title":"MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training","created":"2026-02-25","cited_by_count":6,"area":null},{"id":"2606.04662","title":"Why Muon Outperforms Adam: A Curvature Perspective","created":"2026-06-03","cited_by_count":6,"area":null},{"id":"2502.06806","title":"Logits are All We Need to Adapt Closed Models","created":"2025-02-03","cited_by_count":5,"area":null},{"id":"2505.23737","title":"On the Convergence Analysis of Muon","created":"2025-05-29","cited_by_count":5,"area":null},{"id":"2509.24218","title":"Conda: Column-Normalized Adam for Training Large Language Models Faster","created":"2025-09-29","cited_by_count":5,"area":null},{"id":"2605.03769","title":"Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer","created":"2026-05-05","cited_by_count":5,"area":null},{"id":"2605.06654","title":"Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less","created":"2026-05-07","cited_by_count":5,"area":null},{"id":"2410.05052","title":"Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes","created":"2024-10-07","cited_by_count":4,"area":null},{"id":"2412.21124","title":"Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism","created":"2024-12-30","cited_by_count":4,"area":null},{"id":"2505.11564","title":"HessFormer: Hessians at Foundation Scale","created":"2025-05-16","cited_by_count":4,"area":"moe"},{"id":"2602.06880","title":"Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization","created":"2026-02-06","cited_by_count":4,"area":null},{"id":"2603.09581","title":"Towards Understanding Adam Convergence on Highly Degenerate Polynomials","created":"2026-03-10","cited_by_count":4,"area":null},{"id":"2604.09258","title":"Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima","created":"2026-04-10","cited_by_count":4,"area":null},{"id":"2502.18153","title":"SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation","created":"2025-02-25","cited_by_count":3,"area":null},{"id":"2503.01843","title":"When Can You Get Away with Low Memory Adam?","created":"2025-03-03","cited_by_count":3,"area":null},{"id":"2510.00419","title":"Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs","created":"2025-10-01","cited_by_count":3,"area":null},{"id":"2510.09103","title":"AdaPM: a Partial Momentum Algorithm for LLM Training","created":"2025-10-10","cited_by_count":3,"area":null},{"id":"2602.00334","title":"Adaptive Momentum and Nonlinear Damping for Neural Network Training","created":"2026-01-30","cited_by_count":3,"area":null},{"id":"2602.07145","title":"Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate","created":"2026-02-06","cited_by_count":3,"area":null},{"id":"2603.06922","title":"NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks","created":"2026-03-06","cited_by_count":3,"area":null},{"id":"2605.22297","title":"One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs","created":"2026-05-21","cited_by_count":3,"area":null},{"id":"2606.09658","title":"Muon Learns More Robust and Transferable Features than Adam","created":"2026-06-08","cited_by_count":3,"area":null},{"id":"2501.19099","title":"Elucidating Subspace Perturbation in Zeroth-Order Optimization: Theory and Practice at Scale","created":"2025-01-31","cited_by_count":2,"area":null},{"id":"2510.26303","title":"Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime","created":"2025-10-30","cited_by_count":2,"area":null},{"id":"2602.03515","title":"Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation","created":"2026-02-03","cited_by_count":2,"area":null},{"id":"2604.19740","title":"Generalization at the Edge of Stability","created":"2026-04-21","cited_by_count":2,"area":null},{"id":"2605.00650","title":"AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments","created":"2026-05-01","cited_by_count":2,"area":null},{"id":"2605.03869","title":"On Adaptivity in Zeroth-Order Optimization","created":"2026-05-05","cited_by_count":2,"area":null},{"id":"2605.17787","title":"Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates","created":"2026-05-18","cited_by_count":2,"area":null},{"id":"2607.01083","title":"Staleness-Learning Rate Scaling Laws for Asynchronous RLHF","created":"2026-07-01","cited_by_count":2,"area":"alignment"},{"id":"2405.00252","title":"Q-Newton: Hybrid Quantum-Classical Scheduling for Accelerating Neural Network Training with Newton's Gradient Descent","created":"2024-04-30","cited_by_count":1,"area":null},{"id":"2410.24050","title":"A Mechanistic Study of Transformers Training Dynamics","created":"2024-10-31","cited_by_count":1,"area":null},{"id":"2411.06770","title":"Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis","created":"2024-11-11","cited_by_count":1,"area":null},{"id":"2501.06954","title":"A Hessian-informed hyperparameter optimization for differential learning rate","created":"2025-01-12","cited_by_count":1,"area":null},{"id":"2504.17618","title":"The effects of Hessian eigenvalue spectral density type on the applicability of Hessian analysis to generalization capability assessment of neural networks","created":"2025-04-24","cited_by_count":1,"area":null},{"id":"2512.08217","title":"Correction of Decoupled Weight Decay","created":"2025-12-09","cited_by_count":1,"area":null},{"id":"2601.18420","title":"Gradient Regularized Natural Gradients","created":"2026-01-26","cited_by_count":1,"area":null},{"id":"2601.21626","title":"HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning","created":"2026-01-29","cited_by_count":1,"area":null},{"id":"2602.19945","title":"DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models","created":"2026-02-23","cited_by_count":1,"area":null},{"id":"2603.02649","title":"HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization","created":"2026-03-03","cited_by_count":1,"area":null},{"id":"2604.02505","title":"Optimal Projection-Free Adaptive SGD for Matrix Optimization","created":"2026-04-02","cited_by_count":1,"area":null},{"id":"2604.09437","title":"AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning","created":"2026-04-10","cited_by_count":1,"area":null},{"id":"2605.04269","title":"Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization","created":"2026-05-05","cited_by_count":1,"area":null},{"id":"2605.18870","title":"Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows","created":"2026-05-15","cited_by_count":1,"area":null},{"id":"2606.14259","title":"Beyond a Single Explanation of the Adam--SGD Gap","created":"2026-06-12","cited_by_count":1,"area":null},{"id":"2606.15702","title":"Schattor: Schatten-family methods for deep learning optimization","created":"2026-06-14","cited_by_count":1,"area":null},{"id":"2607.14536","title":"Muse: Representation Geometry of Muon Beyond Normalized Momentum","created":"2026-07-16","cited_by_count":1,"area":null},{"id":"2607.22929","title":"Distribution-Specific Curvature Control with Finite-Sample Guarantees for Open-Weight Safety","created":"2026-07-24","cited_by_count":1,"area":null},{"id":"2608.05088","title":"MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning","created":"2026-08-05","cited_by_count":1,"area":null},{"id":"2608.22129","title":"Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence","created":"2026-08-22","cited_by_count":1,"area":null}],"next_cursor":"100"}