{"list":"other","total":60,"rows":[{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2106.09685","title":"LoRA: Low-Rank Adaptation of Large Language Models","created":"2021-06-17","cited_by_count":14380},{"id":"2302.13971","title":"LLaMA: Open and Efficient Foundation Language Models","created":"2023-02-27","cited_by_count":14132},{"id":"1607.06450","title":"Layer Normalization","created":"2016-07-21","cited_by_count":5669},{"id":"2205.01068","title":"OPT: Open Pre-trained Transformer Language Models","created":"2022-05-02","cited_by_count":3512},{"id":"2309.16609","title":"Qwen Technical Report","created":"2023-09-28","cited_by_count":3344},{"id":"1910.07467","title":"Root Mean Square Layer Normalization","created":"2019-10-16","cited_by_count":1480},{"id":"1804.04235","title":"Adafactor: Adaptive Learning Rates with Sublinear Memory Cost","created":"2018-04-11","cited_by_count":905},{"id":"1503.05671","title":"Optimizing Neural Networks with Kronecker-factored Approximate Curvature","created":"2015-03-19","cited_by_count":796},{"id":"1904.00962","title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","created":"2019-04-01","cited_by_count":674},{"id":"2401.02954","title":"DeepSeek LLM: Scaling Open-Source Language Models with Longtermism","created":"2024-01-05","cited_by_count":642},{"id":"1802.04434","title":"signSGD: Compressed Optimisation for Non-Convex Problems","created":"2018-02-13","cited_by_count":571},{"id":"2403.03507","title":"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection","created":"2024-03-06","cited_by_count":463},{"id":"2302.06675","title":"Symbolic Discovery of Optimization Algorithms","created":"2023-02-13","cited_by_count":434},{"id":"1802.09568","title":"Shampoo: Preconditioned Stochastic Tensor Optimization","created":"2018-02-26","cited_by_count":426},{"id":"2205.05198","title":"Reducing Activation Recomputation in Large Transformer Models","created":"2022-05-10","cited_by_count":302},{"id":"1711.04623","title":"Three Factors Influencing Minima in SGD","created":"2017-11-13","cited_by_count":273},{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265},{"id":"2307.05695","title":"ReLoRA: High-Rank Training Through Low-Rank Updates","created":"2023-07-11","cited_by_count":204},{"id":"2409.20325","title":"Old Optimizer, New Norm: An Anthology","created":"2024-09-30","cited_by_count":193},{"id":"2409.11321","title":"SOAP: Improving and Stabilizing Shampoo using Adam","created":"2024-09-17","cited_by_count":192},{"id":"1806.03884","title":"Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis","created":"2018-06-11","cited_by_count":146},{"id":"2002.03305","title":"Momentum Improves Normalized SGD","created":"2020-02-09","cited_by_count":139},{"id":"2406.16793","title":"Adam-mini: Use Fewer Learning Rates To Gain More","created":"2024-06-24","cited_by_count":137},{"id":"2306.09782","title":"Full Parameter Fine-tuning for Large Language Models with Limited Resources","created":"2023-06-16","cited_by_count":134},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114},{"id":"2402.03293","title":"Flora: Low-Rank Adapters Are Secretly Gradient Compressors","created":"2024-02-05","cited_by_count":108},{"id":"1904.03441","title":"Iterative Normalization: Beyond Standardization towards Efficient Whitening","created":"2019-04-06","cited_by_count":92},{"id":"1804.08450","title":"Decorrelated Batch Normalization","created":"2018-04-23","cited_by_count":83},{"id":"2208.11195","title":"Robustness to Unbounded Smoothness of Generalized SignSGD","created":"2022-08-23","cited_by_count":81},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80},{"id":"2309.06497","title":"A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale","created":"2023-09-12","cited_by_count":77},{"id":"1712.01034","title":"Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization","created":"2017-12-04","cited_by_count":65},{"id":"2304.09871","title":"A Theory on Adam Instability in Large-Scale Machine Learning","created":"2023-04-19","cited_by_count":56},{"id":"2409.03137","title":"The AdEMAMix Optimizer: Better, Faster, Older","created":"2024-09-05","cited_by_count":54},{"id":"2412.05270","title":"APOLLO: SGD-like Memory, AdamW-level Performance","created":"2024-12-06","cited_by_count":54},{"id":"2310.00535","title":"JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention","created":"2023-10-01","cited_by_count":52},{"id":"1512.04202","title":"Preconditioned Stochastic Gradient Descent","created":"2015-12-14","cited_by_count":50},{"id":"2311.00636","title":"Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures","created":"2023-11-01","cited_by_count":49},{"id":"2410.01623","title":"Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?","created":"2024-10-02","cited_by_count":43},{"id":"2307.06440","title":"No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models","created":"2023-07-12","cited_by_count":41},{"id":"2407.07972","title":"Deconstructing What Makes a Good Optimizer for Language Models","created":"2024-07-10","cited_by_count":41},{"id":"2407.08296","title":"Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients","created":"2024-07-11","cited_by_count":38},{"id":"2209.13569","title":"Exploring Low Rank Training of Deep Neural Networks","created":"2022-09-27","cited_by_count":30},{"id":"1810.13108","title":"A general system of differential equations to model first order adaptive algorithms","created":"2018-10-31","cited_by_count":28},{"id":"2202.07052","title":"Orthogonalising gradients to speed up neural network optimisation","created":"2022-02-14","cited_by_count":27},{"id":"2402.03496","title":"Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective","created":"2024-02-05","cited_by_count":25},{"id":"2211.02254","title":"How Does Adaptive Optimization Impact Local Neural Network Geometry?","created":"2022-11-04","cited_by_count":24},{"id":"2405.12807","title":"FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information","created":"2024-05-21","cited_by_count":18},{"id":"2405.18144","title":"4-bit Shampoo for Memory-Efficient Network Training","created":"2024-05-28","cited_by_count":18},{"id":"2201.12543","title":"Fast Differentiable Matrix Square Root and Inverse Square Root","created":"2022-01-29","cited_by_count":16},{"id":"2402.04553","title":"Curvature-Informed SGD via General Purpose Lie-Group Preconditioners","created":"2024-02-07","cited_by_count":16},{"id":"2011.10741","title":"A Trace-restricted Kronecker-Factored Approximation to Natural Gradient","created":"2020-11-21","cited_by_count":11},{"id":"2201.10285","title":"Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decomposition","created":"2022-01-25","cited_by_count":9},{"id":"2412.11768","title":"No More Adam: Learning Rate Scaling at Initialization is All You Need","created":"2024-12-16","cited_by_count":8},{"id":"2011.13609","title":"Eigenvalue-corrected Natural Gradient Based on a New Approximation","created":"2020-11-27","cited_by_count":4},{"id":"2305.19416","title":"KrADagrad: Kronecker Approximation-Domination Gradient Preconditioned Stochastic Optimization","created":"2023-05-30","cited_by_count":3}],"next_cursor":null}