{"list":"other","total":55,"rows":[{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2204.02311","title":"PaLM: Scaling Language Modeling with Pathways","created":"2022-04-05","cited_by_count":5326},{"id":"1701.06538","title":"Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer","created":"2017-01-23","cited_by_count":3509},{"id":"2101.03961","title":"Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity","created":"2021-01-11","cited_by_count":3073},{"id":"2101.00027","title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling","created":"2020-12-31","cited_by_count":2411},{"id":"2006.16668","title":"GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding","created":"2020-06-30","cited_by_count":1631},{"id":"2010.01412","title":"Sharpness-Aware Minimization for Efficiently Improving Generalization","created":"2020-10-03","cited_by_count":1417},{"id":"1609.04836","title":"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima","created":"2016-09-15","cited_by_count":1352},{"id":"1606.04838","title":"Optimization Methods for Large-Scale Machine Learning","created":"2016-06-15","cited_by_count":1150},{"id":"2002.04745","title":"On Layer Normalization in the Transformer Architecture","created":"2020-02-12","cited_by_count":922},{"id":"2112.06905","title":"GLaM: Efficient Scaling of Language Models with Mixture-of-Experts","created":"2021-12-13","cited_by_count":897},{"id":"1503.05671","title":"Optimizing Neural Networks with Kronecker-factored Approximate Curvature","created":"2015-03-19","cited_by_count":796},{"id":"2212.07677","title":"Transformers learn in-context by gradient descent","created":"2022-12-15","cited_by_count":732},{"id":"2208.01066","title":"What Can Transformers Learn In-Context? A Case Study of Simple Function Classes","created":"2022-08-01","cited_by_count":715},{"id":"2211.15661","title":"What learning algorithm is in-context learning? Investigations with linear models","created":"2022-11-28","cited_by_count":684},{"id":"2403.03507","title":"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection","created":"2024-03-06","cited_by_count":463},{"id":"1802.09568","title":"Shampoo: Preconditioned Stochastic Tensor Optimization","created":"2018-02-26","cited_by_count":426},{"id":"2110.02861","title":"8-bit Optimizers via Block-wise Quantization","created":"2021-10-06","cited_by_count":403},{"id":"2202.08906","title":"ST-MoE: Designing Stable and Transferable Sparse Expert Models","created":"2022-02-17","cited_by_count":399},{"id":"1905.11881","title":"Why gradient clipping accelerates training: A theoretical justification for adaptivity","created":"2019-05-28","cited_by_count":341},{"id":"2102.11600","title":"ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks","created":"2021-02-23","cited_by_count":293},{"id":"1406.2572","title":"Identifying and attacking the saddle point problem in high-dimensional non-convex optimization","created":"2014-06-10","cited_by_count":289},{"id":"1711.04623","title":"Three Factors Influencing Minima in SGD","created":"2017-11-13","cited_by_count":273},{"id":"1502.04759","title":"Coordinate Descent Algorithms","created":"2015-02-17","cited_by_count":219},{"id":"2006.00719","title":"ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning","created":"2020-06-01","cited_by_count":202},{"id":"2409.11321","title":"SOAP: Improving and Stabilizing Shampoo using Adam","created":"2024-09-17","cited_by_count":192},{"id":"2203.00555","title":"DeepNet: Scaling Transformers to 1,000 Layers","created":"2022-03-01","cited_by_count":178},{"id":"1910.05895","title":"Transformers without Tears: Improving the Normalization of Self-Attention","created":"2019-10-14","cited_by_count":153},{"id":"1511.06807","title":"Adding Gradient Noise Improves Learning for Very Deep Networks","created":"2015-11-21","cited_by_count":145},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124},{"id":"2402.11592","title":"Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark","created":"2024-02-18","cited_by_count":114},{"id":"2310.05914","title":"NEFTune: Noisy Embeddings Improve Instruction Finetuning","created":"2023-10-09","cited_by_count":100},{"id":"2310.01082","title":"Linear attention is (maybe) all you need (to understand transformer optimization)","created":"2023-10-02","cited_by_count":98},{"id":"2403.17919","title":"LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning","created":"2024-03-26","cited_by_count":89},{"id":"2208.11195","title":"Robustness to Unbounded Smoothness of Generalized SignSGD","created":"2022-08-23","cited_by_count":81},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80},{"id":"2002.12915","title":"The Implicit and Explicit Regularization Effects of Dropout","created":"2020-02-28","cited_by_count":71},{"id":"2004.10397","title":"A Framework for Evaluating Gradient Leakage Attacks in Federated Learning","created":"2020-04-22","cited_by_count":64},{"id":"2412.05270","title":"APOLLO: SGD-like Memory, AdamW-level Performance","created":"2024-12-06","cited_by_count":54},{"id":"2306.00204","title":"Toward Understanding Why Adam Converges Faster Than SGD for Transformers","created":"2023-05-31","cited_by_count":53},{"id":"2312.16903","title":"Spike No More: Stabilizing the Pre-training of Large Language Models","created":"2023-12-28","cited_by_count":52},{"id":"2411.16085","title":"Cautious Optimizers: Improving Training with One Line of Code","created":"2024-11-25","cited_by_count":47},{"id":"2505.21829","title":"In Search of Adam's Secret Sauce","created":"2025-05-27","cited_by_count":40},{"id":"2501.06842","title":"SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training","created":"2025-01-12","cited_by_count":28},{"id":"2306.04226","title":"Normalization Layers Are All That Sharpness-Aware Minimization Needs","created":"2023-06-07","cited_by_count":27},{"id":"2404.02827","title":"BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models","created":"2024-04-03","cited_by_count":27},{"id":"2207.05952","title":"Implicit regularization of dropout","created":"2022-07-13","cited_by_count":25},{"id":"2211.02254","title":"How Does Adaptive Optimization Impact Local Neural Network Geometry?","created":"2022-11-04","cited_by_count":24},{"id":"1806.09777","title":"On the Implicit Bias of Dropout","created":"2018-06-26","cited_by_count":22},{"id":"2002.04839","title":"LaProp: Separating Momentum and Adaptivity in Adam","created":"2020-02-12","cited_by_count":20},{"id":"1712.08859","title":"Let's Make Block Coordinate Descent Converge Faster: Faster Greedy Rules, Message-Passing, Active-Set Complexity, and Superlinear Convergence","created":"2017-12-23","cited_by_count":17},{"id":"2004.05859","title":"Regularizing Meta-Learning via Gradient Dropout","created":"2020-04-13","cited_by_count":15},{"id":"2410.15352","title":"CompAct: Compressed Activations for Memory-Efficient LLM Training","created":"2024-10-20","cited_by_count":15},{"id":"2410.10986","title":"What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis","created":"2024-10-14","cited_by_count":9},{"id":"2210.05607","title":"Divergence Results and Convergence of a Variance Reduced Version of ADAM","created":"2022-10-11","cited_by_count":8}],"next_cursor":null}