{"list":"other","total":61,"rows":[{"id":"1810.04805","title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","created":"2018-10-11","cited_by_count":40399},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"2412.15115","title":"Qwen2.5 Technical Report","created":"2024-12-19","cited_by_count":4942},{"id":"2404.14219","title":"Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone","created":"2024-04-22","cited_by_count":1917},{"id":"2002.05202","title":"GLU Variants Improve Transformer","created":"2020-02-12","cited_by_count":1804},{"id":"1906.02243","title":"Energy and Policy Considerations for Deep Learning in NLP","created":"2019-06-05","cited_by_count":1254},{"id":"2112.11446","title":"Scaling Language Models: Methods, Analysis & Insights from Training Gopher","created":"2021-12-08","cited_by_count":1202},{"id":"2108.12409","title":"Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation","created":"2021-08-27","cited_by_count":1103},{"id":"2404.06395","title":"MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies","created":"2024-04-09","cited_by_count":723},{"id":"1904.00962","title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","created":"2019-04-01","cited_by_count":674},{"id":"2401.02954","title":"DeepSeek LLM: Scaling Open-Source Language Models with Longtermism","created":"2024-01-05","cited_by_count":642},{"id":"2104.10350","title":"Carbon Emissions and Large Neural Network Training","created":"2021-04-21","cited_by_count":567},{"id":"2201.11990","title":"Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model","created":"2022-01-28","cited_by_count":559},{"id":"1708.03888","title":"Large Batch Training of Convolutional Networks","created":"2017-08-13","cited_by_count":535},{"id":"2311.16867","title":"The Falcon Series of Open Language Models","created":"2023-11-28","cited_by_count":535},{"id":"1802.09568","title":"Shampoo: Preconditioned Stochastic Tensor Optimization","created":"2018-02-26","cited_by_count":426},{"id":"1711.00489","title":"Don't Decay the Learning Rate, Increase the Batch Size","created":"2017-11-01","cited_by_count":320},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"1705.08741","title":"Train longer, generalize better: closing the generalization gap in large batch training of neural networks","created":"2017-05-24","cited_by_count":279},{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265},{"id":"1404.5997","title":"One weird trick for parallelizing convolutional neural networks","created":"2014-04-23","cited_by_count":264},{"id":"1811.03600","title":"Measuring the Effects of Data Parallelism on Neural Network Training","created":"2018-11-08","cited_by_count":193},{"id":"2409.11321","title":"SOAP: Improving and Stabilizing Shampoo using Adam","created":"2024-09-17","cited_by_count":192},{"id":"2011.14522","title":"Feature Learning in Infinite-Width Neural Networks","created":"2020-11-30","cited_by_count":190},{"id":"2309.14322","title":"Small-scale proxies for large-scale Transformer training instabilities","created":"2023-09-25","cited_by_count":183},{"id":"2402.07871","title":"Scaling Laws for Fine-Grained Mixture of Experts","created":"2024-02-12","cited_by_count":173},{"id":"2405.18392","title":"Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations","created":"2024-05-28","cited_by_count":170},{"id":"1712.06559","title":"The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning","created":"2017-12-18","cited_by_count":150},{"id":"1702.05800","title":"Revisiting Distributed Synchronous SGD","created":"2017-02-19","cited_by_count":142},{"id":"1710.06451","title":"A Bayesian Perspective on Generalization and Stochastic Gradient Descent","created":"2017-10-17","cited_by_count":135},{"id":"1706.05350","title":"L2 Regularization versus Batch and Weight Normalization","created":"2017-06-16","cited_by_count":124},{"id":"1910.07454","title":"An Exponential Learning Rate Schedule for Deep Learning","created":"2019-10-16","cited_by_count":121},{"id":"1907.04164","title":"Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model","created":"2019-07-09","cited_by_count":117},{"id":"1810.12281","title":"Three Mechanisms of Weight Decay Regularization","created":"2018-10-29","cited_by_count":113},{"id":"2403.08540","title":"Language models scale reliably with over-training and on downstream tasks","created":"2024-03-13","cited_by_count":113},{"id":"2304.03208","title":"Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster","created":"2023-04-06","cited_by_count":109},{"id":"2205.10287","title":"On the SDEs and Scaling Rules for Adaptive Gradient Algorithms","created":"2022-05-20","cited_by_count":104},{"id":"2411.04330","title":"Scaling Laws for Precision","created":"2024-11-07","cited_by_count":95},{"id":"2406.19146","title":"Resolving Discrepancies in Compute-Optimal Scaling of Language Models","created":"2024-06-27","cited_by_count":88},{"id":"2410.05192","title":"Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective","created":"2024-10-07","cited_by_count":81},{"id":"2505.01618","title":"Don't be lazy: CompleteP enables compute-efficient deep transformers","created":"2025-05-02","cited_by_count":78},{"id":"2410.21676","title":"How Does Critical Batch Size Scale in Pre-training?","created":"2024-10-29","cited_by_count":73},{"id":"2310.04415","title":"Why Do We Need Weight Decay in Modern Deep Learning?","created":"2023-10-06","cited_by_count":72},{"id":"2404.10102","title":"Chinchilla Scaling: A replication attempt","created":"2024-04-15","cited_by_count":72},{"id":"2308.16149","title":"Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models","created":"2023-08-30","cited_by_count":69},{"id":"1803.01814","title":"Norm matters: efficient and accurate normalization schemes in deep networks","created":"2018-03-05","cited_by_count":62},{"id":"2305.17212","title":"Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks","created":"2023-05-26","cited_by_count":53},{"id":"2405.13698","title":"How to set AdamW's weight decay as you scale model and dataset size","created":"2024-05-22","cited_by_count":43},{"id":"2409.19913","title":"Scaling Optimal LR Across Token Horizons","created":"2024-09-30","cited_by_count":40},{"id":"2502.15938","title":"Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs","created":"2025-02-21","cited_by_count":35},{"id":"2503.04715","title":"Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining","created":"2025-03-06","cited_by_count":34},{"id":"1905.05894","title":"Online Normalization for Training Neural Networks","created":"2019-05-15","cited_by_count":29},{"id":"2408.13359","title":"Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler","created":"2024-08-23","cited_by_count":28},{"id":"1811.12941","title":"On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent","created":"2018-11-30","cited_by_count":25},{"id":"2405.14578","title":"Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling","created":"2024-05-23","cited_by_count":22},{"id":"2410.23922","title":"Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training","created":"2024-10-31","cited_by_count":22},{"id":"2309.11568","title":"BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model","created":"2023-09-20","cited_by_count":19},{"id":"2410.05838","title":"Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit","created":"2024-10-08","cited_by_count":9},{"id":"2406.18922","title":"Time Matters: Scaling Laws for Any Budget","created":"2024-06-27","cited_by_count":4}],"next_cursor":null}