{"list":"other","total":65,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"1512.03385","title":"Deep Residual Learning for Image Recognition","created":"2015-12-10","cited_by_count":52096},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2307.09288","title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","created":"2023-07-18","cited_by_count":13028},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"1502.01852","title":"Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification","created":"2015-02-06","cited_by_count":3611},{"id":"1605.07146","title":"Wide Residual Networks","created":"2016-05-23","cited_by_count":3070},{"id":"2101.00027","title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling","created":"2020-12-31","cited_by_count":2411},{"id":"2002.05202","title":"GLU Variants Improve Transformer","created":"2020-02-12","cited_by_count":1804},{"id":"2211.17192","title":"Fast Inference from Transformers via Speculative Decoding","created":"2022-11-30","cited_by_count":1653},{"id":"1706.02677","title":"Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour","created":"2017-06-08","cited_by_count":1492},{"id":"2112.11446","title":"Scaling Language Models: Methods, Analysis & Insights from Training Gopher","created":"2021-12-08","cited_by_count":1202},{"id":"1606.04838","title":"Optimization Methods for Large-Scale Machine Learning","created":"2016-06-15","cited_by_count":1150},{"id":"2108.12409","title":"Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation","created":"2021-08-27","cited_by_count":1103},{"id":"2106.04560","title":"Scaling Vision Transformers","created":"2021-06-08","cited_by_count":999},{"id":"2002.04745","title":"On Layer Normalization in the Transformer Architecture","created":"2020-02-12","cited_by_count":922},{"id":"2403.08295","title":"Gemma: Open Models Based on Gemini Research and Technology","created":"2024-03-13","cited_by_count":916},{"id":"1908.03265","title":"On the Variance of the Adaptive Learning Rate and Beyond","created":"2019-08-08","cited_by_count":772},{"id":"2404.06395","title":"MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies","created":"2024-04-09","cited_by_count":723},{"id":"1904.00962","title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","created":"2019-04-01","cited_by_count":674},{"id":"2401.02954","title":"DeepSeek LLM: Scaling Open-Source Language Models with Longtermism","created":"2024-01-05","cited_by_count":642},{"id":"1506.01186","title":"Cyclical Learning Rates for Training Neural Networks","created":"2015-06-03","cited_by_count":629},{"id":"2309.05463","title":"Textbooks Are All You Need II: phi-1.5 technical report","created":"2023-09-11","cited_by_count":565},{"id":"1308.0850","title":"Generating Sequences With Recurrent Neural Networks","created":"2013-08-04","cited_by_count":554},{"id":"2311.16867","title":"The Falcon Series of Open Language Models","created":"2023-11-28","cited_by_count":535},{"id":"2102.00554","title":"Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks","created":"2021-01-31","cited_by_count":527},{"id":"2010.14701","title":"Scaling Laws for Autoregressive Generative Modeling","created":"2020-10-28","cited_by_count":522},{"id":"1802.09568","title":"Shampoo: Preconditioned Stochastic Tensor Optimization","created":"2018-02-26","cited_by_count":426},{"id":"2305.16264","title":"Scaling Data-Constrained Language Models","created":"2023-05-25","cited_by_count":403},{"id":"1711.00489","title":"Don't Decay the Learning Rate, Increase the Batch Size","created":"2017-11-01","cited_by_count":320},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"1705.08741","title":"Train longer, generalize better: closing the generalization gap in large batch training of neural networks","created":"2017-05-24","cited_by_count":279},{"id":"1404.5997","title":"One weird trick for parallelizing convolutional neural networks","created":"2014-04-23","cited_by_count":264},{"id":"1811.03600","title":"Measuring the Effects of Data Parallelism on Neural Network Training","created":"2018-11-08","cited_by_count":193},{"id":"2011.14522","title":"Feature Learning in Infinite-Width Neural Networks","created":"2020-11-30","cited_by_count":190},{"id":"2309.14322","title":"Small-scale proxies for large-scale Transformer training instabilities","created":"2023-09-25","cited_by_count":183},{"id":"2405.18392","title":"Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations","created":"2024-05-28","cited_by_count":170},{"id":"2405.15682","title":"The Road Less Scheduled","created":"2024-05-24","cited_by_count":163},{"id":"1702.05800","title":"Revisiting Distributed Synchronous SGD","created":"2017-02-19","cited_by_count":142},{"id":"2403.08763","title":"Simple and Scalable Strategies to Continually Pre-train Large Language Models","created":"2024-03-13","cited_by_count":127},{"id":"2007.01547","title":"Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers","created":"2020-07-03","cited_by_count":119},{"id":"2304.03208","title":"Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster","created":"2023-04-06","cited_by_count":109},{"id":"2205.10287","title":"On the SDEs and Scaling Rules for Adaptive Gradient Algorithms","created":"2022-05-20","cited_by_count":104},{"id":"2406.19146","title":"Resolving Discrepancies in Compute-Optimal Scaling of Language Models","created":"2024-06-27","cited_by_count":88},{"id":"1802.08770","title":"A Walk with SGD","created":"2018-02-24","cited_by_count":78},{"id":"2410.21676","title":"How Does Critical Batch Size Scale in Pre-training?","created":"2024-10-29","cited_by_count":73},{"id":"2310.04415","title":"Why Do We Need Weight Decay in Modern Deep Learning?","created":"2023-10-06","cited_by_count":72},{"id":"2308.16149","title":"Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models","created":"2023-08-30","cited_by_count":69},{"id":"1904.12838","title":"The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares","created":"2019-04-29","cited_by_count":68},{"id":"2010.02916","title":"Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate","created":"2020-10-06","cited_by_count":59},{"id":"2404.07413","title":"JetMoE: Reaching Llama2 Performance with 0.1M Dollars","created":"2024-04-11","cited_by_count":57},{"id":"2405.13698","title":"How to set AdamW's weight decay as you scale model and dataset size","created":"2024-05-22","cited_by_count":43},{"id":"2409.19913","title":"Scaling Optimal LR Across Token Horizons","created":"2024-09-30","cited_by_count":40},{"id":"1905.04753","title":"Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints","created":"2019-05-12","cited_by_count":37},{"id":"2306.03241","title":"Early Weight Averaging meets High Learning Rates for LLM Pre-training","created":"2023-06-05","cited_by_count":37},{"id":"2408.11029","title":"Scaling Law with Learning Rate Annealing","created":"2024-08-20","cited_by_count":29},{"id":"2408.13359","title":"Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler","created":"2024-08-23","cited_by_count":28},{"id":"2307.13813","title":"How to Scale Your EMA","created":"2023-07-25","cited_by_count":24},{"id":"2410.23922","title":"Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training","created":"2024-10-31","cited_by_count":22},{"id":"2301.02312","title":"Training trajectories, mini-batch losses and the curious role of the learning rate","created":"2023-01-05","cited_by_count":21},{"id":"2405.15743","title":"Sparse maximal update parameterization: A holistic approach to sparse training dynamics","created":"2024-05-24","cited_by_count":16},{"id":"2410.05838","title":"Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit","created":"2024-10-08","cited_by_count":9},{"id":"2411.00999","title":"Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers","created":"2024-11-01","cited_by_count":7}],"next_cursor":null}