{"list":"other","total":51,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"1412.6980","title":"Adam: A Method for Stochastic Optimization","created":"2014-12-22","cited_by_count":37240},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2302.13971","title":"LLaMA: Open and Efficient Foundation Language Models","created":"2023-02-27","cited_by_count":14132},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"1608.03983","title":"SGDR: Stochastic Gradient Descent with Warm Restarts","created":"2016-08-13","cited_by_count":5045},{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019},{"id":"1909.08053","title":"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism","created":"2019-09-17","cited_by_count":1868},{"id":"1706.02677","title":"Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour","created":"2017-06-08","cited_by_count":1492},{"id":"1803.05407","title":"Averaging Weights Leads to Wider Optima and Better Generalization","created":"2018-03-14","cited_by_count":1209},{"id":"2106.04560","title":"Scaling Vision Transformers","created":"2021-06-08","cited_by_count":999},{"id":"2404.06395","title":"MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies","created":"2024-04-09","cited_by_count":723},{"id":"2302.05442","title":"Scaling Vision Transformers to 22 Billion Parameters","created":"2023-02-10","cited_by_count":694},{"id":"2401.02954","title":"DeepSeek LLM: Scaling Open-Source Language Models with Longtermism","created":"2024-01-05","cited_by_count":642},{"id":"2402.00838","title":"OLMo: Accelerating the Science of Language Models","created":"2024-02-01","cited_by_count":613},{"id":"1712.00409","title":"Deep Learning Scaling is Predictable, Empirically","created":"2017-12-01","cited_by_count":582},{"id":"2201.11990","title":"Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model","created":"2022-01-28","cited_by_count":559},{"id":"1708.03888","title":"Large Batch Training of Convolutional Networks","created":"2017-08-13","cited_by_count":535},{"id":"2010.14701","title":"Scaling Laws for Autoregressive Generative Modeling","created":"2020-10-28","cited_by_count":522},{"id":"2305.16264","title":"Scaling Data-Constrained Language Models","created":"2023-05-25","cited_by_count":403},{"id":"2312.02696","title":"Analyzing and Improving the Training Dynamics of Diffusion Models","created":"2023-12-05","cited_by_count":402},{"id":"1711.00489","title":"Don't Decay the Learning Rate, Increase the Batch Size","created":"2017-11-01","cited_by_count":320},{"id":"2203.03466","title":"Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer","created":"2022-03-07","cited_by_count":288},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"1811.03600","title":"Measuring the Effects of Data Parallelism on Neural Network Training","created":"2018-11-08","cited_by_count":193},{"id":"2309.14322","title":"Small-scale proxies for large-scale Transformer training instabilities","created":"2023-09-25","cited_by_count":183},{"id":"2405.15682","title":"The Road Less Scheduled","created":"2024-05-24","cited_by_count":163},{"id":"2303.06296","title":"Stabilizing Transformer Training by Preventing Attention Entropy Collapse","created":"2023-03-11","cited_by_count":161},{"id":"1712.06559","title":"The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning","created":"2017-12-18","cited_by_count":150},{"id":"1710.06451","title":"A Bayesian Perspective on Generalization and Stochastic Gradient Descent","created":"2017-10-17","cited_by_count":135},{"id":"1907.04164","title":"Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model","created":"2019-07-09","cited_by_count":117},{"id":"2208.09632","title":"Adam Can Converge Without Any Modification On Update Rules","created":"2022-08-20","cited_by_count":114},{"id":"2402.01092","title":"A Dynamical Model of Neural Scaling Laws","created":"2024-02-02","cited_by_count":111},{"id":"2304.03208","title":"Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster","created":"2023-04-06","cited_by_count":109},{"id":"2205.09653","title":"Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks","created":"2022-05-19","cited_by_count":105},{"id":"2310.02244","title":"Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks","created":"2023-10-03","cited_by_count":97},{"id":"2406.19146","title":"Resolving Discrepancies in Compute-Optimal Scaling of Language Models","created":"2024-06-27","cited_by_count":88},{"id":"2102.12470","title":"On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)","created":"2021-02-24","cited_by_count":73},{"id":"2404.10102","title":"Chinchilla Scaling: A replication attempt","created":"2024-04-15","cited_by_count":72},{"id":"2103.12692","title":"Benign Overfitting of Constant-Stepsize SGD for Linear Regression","created":"2021-03-23","cited_by_count":71},{"id":"2405.15074","title":"4+3 Phases of Compute-Optimal Neural Scaling Laws","created":"2024-05-23","cited_by_count":68},{"id":"2309.16620","title":"Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit","created":"2023-09-28","cited_by_count":66},{"id":"2406.08466","title":"Scaling Laws in Linear Regression: Compute, Parameters, and Data","created":"2024-06-12","cited_by_count":57},{"id":"2305.17212","title":"Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks","created":"2023-05-26","cited_by_count":53},{"id":"2305.18411","title":"Feature-Learning Networks Are Consistent Across Widths At Realistic Scales","created":"2023-05-28","cited_by_count":41},{"id":"2405.15712","title":"Infinite Limits of Multi-head Transformer Dynamics","created":"2024-05-24","cited_by_count":39},{"id":"2406.12907","title":"Reconciling Kaplan and Chinchilla Scaling Laws","created":"2024-06-12","cited_by_count":39},{"id":"2110.00641","title":"Batch size-invariance for policy optimization","created":"2021-10-01","cited_by_count":35},{"id":"1811.12941","title":"On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent","created":"2018-11-30","cited_by_count":25},{"id":"2309.11568","title":"BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model","created":"2023-09-20","cited_by_count":19},{"id":"2410.05838","title":"Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limit","created":"2024-10-08","cited_by_count":9}],"next_cursor":null}