{"list":"other","total":66,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"1412.6980","title":"Adam: A Method for Stochastic Optimization","created":"2014-12-22","cited_by_count":37240},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2106.09685","title":"LoRA: Low-Rank Adaptation of Large Language Models","created":"2021-06-17","cited_by_count":14380},{"id":"2302.13971","title":"LLaMA: Open and Efficient Foundation Language Models","created":"2023-02-27","cited_by_count":14132},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"2103.03874","title":"Measuring Mathematical Problem Solving With the MATH Dataset","created":"2021-03-05","cited_by_count":5371},{"id":"2204.02311","title":"PaLM: Scaling Language Modeling with Pathways","created":"2022-04-05","cited_by_count":5326},{"id":"2104.09864","title":"RoFormer: Enhanced Transformer with Rotary Position Embedding","created":"2021-04-20","cited_by_count":4703},{"id":"1606.08415","title":"Gaussian Error Linear Units (GELUs)","created":"2016-06-27","cited_by_count":3502},{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019},{"id":"2211.05100","title":"BLOOM: A 176B-Parameter Open-Access Multilingual Language Model","created":"2022-11-09","cited_by_count":1984},{"id":"2503.19786","title":"Gemma 3 Technical Report","created":"2025-03-25","cited_by_count":1824},{"id":"1706.02677","title":"Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour","created":"2017-06-08","cited_by_count":1492},{"id":"1910.07467","title":"Root Mean Square Layer Normalization","created":"2019-10-16","cited_by_count":1480},{"id":"1609.04836","title":"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima","created":"2016-09-15","cited_by_count":1352},{"id":"2406.17557","title":"The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale","created":"2024-06-25","cited_by_count":1160},{"id":"1804.04235","title":"Adafactor: Adaptive Learning Rates with Sublinear Memory Cost","created":"2018-04-11","cited_by_count":905},{"id":"2210.02414","title":"GLM-130B: An Open Bilingual Pre-trained Model","created":"2022-10-05","cited_by_count":843},{"id":"2302.05442","title":"Scaling Vision Transformers to 22 Billion Parameters","created":"2023-02-10","cited_by_count":694},{"id":"1904.00962","title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","created":"2019-04-01","cited_by_count":674},{"id":"1708.03888","title":"Large Batch Training of Convolutional Networks","created":"2017-08-13","cited_by_count":535},{"id":"2104.08758","title":"Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus","created":"2021-04-18","cited_by_count":492},{"id":"1802.09568","title":"Shampoo: Preconditioned Stochastic Tensor Optimization","created":"2018-02-26","cited_by_count":426},{"id":"1905.13655","title":"Implicit Regularization in Deep Matrix Factorization","created":"2019-05-31","cited_by_count":405},{"id":"2405.09673","title":"LoRA Learns Less and Forgets Less","created":"2024-05-15","cited_by_count":328},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265},{"id":"1503.02101","title":"Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition","created":"2015-03-06","cited_by_count":243},{"id":"2003.02218","title":"The large learning rate phase of deep learning: the catapult mechanism","created":"2020-03-04","cited_by_count":216},{"id":"2304.01433","title":"TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings","created":"2023-04-04","cited_by_count":202},{"id":"1811.03600","title":"Measuring the Effects of Data Parallelism on Neural Network Training","created":"2018-11-08","cited_by_count":193},{"id":"2409.11321","title":"SOAP: Improving and Stabilizing Shampoo using Adam","created":"2024-09-17","cited_by_count":192},{"id":"2101.12176","title":"On the Origin of Implicit Regularization in Stochastic Gradient Descent","created":"2021-01-28","cited_by_count":184},{"id":"2002.09572","title":"The Break-Even Point on Optimization Trajectories of Deep Neural Networks","created":"2020-02-21","cited_by_count":150},{"id":"2009.11162","title":"Implicit Gradient Regularization","created":"2020-09-23","cited_by_count":146},{"id":"1702.05800","title":"Revisiting Distributed Synchronous SGD","created":"2017-02-19","cited_by_count":142},{"id":"1710.06451","title":"A Bayesian Perspective on Generalization and Stochastic Gradient Descent","created":"2017-10-17","cited_by_count":135},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124},{"id":"1907.04164","title":"Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model","created":"2019-07-09","cited_by_count":117},{"id":"2208.09632","title":"Adam Can Converge Without Any Modification On Update Rules","created":"2022-08-20","cited_by_count":114},{"id":"2205.10287","title":"On the SDEs and Scaling Rules for Adaptive Gradient Algorithms","created":"2022-05-20","cited_by_count":104},{"id":"1804.07612","title":"Revisiting Small Batch Training for Deep Neural Networks","created":"2018-04-20","cited_by_count":102},{"id":"1807.05031","title":"On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length","created":"2018-07-13","cited_by_count":88},{"id":"2006.08680","title":"Shape Matters: Understanding the Implicit Bias of the Noise Covariance","created":"2020-06-15","cited_by_count":88},{"id":"2406.19146","title":"Resolving Discrepancies in Compute-Optimal Scaling of Language Models","created":"2024-06-27","cited_by_count":88},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80},{"id":"1705.10412","title":"Gradient Descent Can Take Exponential Time to Escape Saddle Points","created":"2017-05-29","cited_by_count":74},{"id":"2410.21676","title":"How Does Critical Batch Size Scale in Pre-training?","created":"2024-10-29","cited_by_count":73},{"id":"2109.14119","title":"Stochastic Training is Not Necessary for Generalization","created":"2021-09-29","cited_by_count":63},{"id":"2006.09092","title":"Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training","created":"2020-06-16","cited_by_count":50},{"id":"1803.05591","title":"On the insufficiency of existing momentum schemes for Stochastic Optimization","created":"2018-03-15","cited_by_count":49},{"id":"2012.15355","title":"Optimizing Deeper Transformers on Small Datasets","created":"2020-12-30","cited_by_count":42},{"id":"2307.13813","title":"How to Scale Your EMA","created":"2023-07-25","cited_by_count":24},{"id":"2409.15156","title":"Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scaling","created":"2024-09-23","cited_by_count":23},{"id":"2405.14578","title":"Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling","created":"2024-05-23","cited_by_count":22},{"id":"2206.01029","title":"Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions","created":"2022-06-02","cited_by_count":19},{"id":"2306.09000","title":"When and Why Momentum Accelerates SGD:An Empirical Study","created":"2023-06-15","cited_by_count":19},{"id":"2307.02053","title":"Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuning","created":"2023-07-05","cited_by_count":18},{"id":"2307.15196","title":"The Marginal Value of Momentum for Small Learning Rate SGD","created":"2023-07-27","cited_by_count":18},{"id":"2410.04642","title":"The Optimization Landscape of SGD Across the Feature Learning Strength","created":"2024-10-06","cited_by_count":13},{"id":"2306.08590","title":"Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning","created":"2023-06-14","cited_by_count":10},{"id":"1902.00071","title":"Optimal mini-batch and step sizes for SAGA","created":"2019-01-31","cited_by_count":9},{"id":"2410.05838","title":"Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit","created":"2024-10-08","cited_by_count":9},{"id":"2402.18824","title":"Batch size invariant Adam","created":"2024-02-29","cited_by_count":5},{"id":"1910.08222","title":"Improving the convergence of SGD through adaptive batch sizes","created":"2019-10-18","cited_by_count":1}],"next_cursor":null}