{"list":"other","total":62,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2010.11929","title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","created":"2020-10-22","cited_by_count":30186},{"id":"1912.01703","title":"PyTorch: An Imperative Style, High-Performance Deep Learning Library","created":"2019-12-03","cited_by_count":20305},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"1907.11692","title":"RoBERTa: A Robustly Optimized BERT Pretraining Approach","created":"2019-07-26","cited_by_count":13372},{"id":"2307.09288","title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","created":"2023-07-18","cited_by_count":13028},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"2205.01068","title":"OPT: Open Pre-trained Transformer Language Models","created":"2022-05-02","cited_by_count":3512},{"id":"2309.16609","title":"Qwen Technical Report","created":"2023-09-28","cited_by_count":3344},{"id":"2407.10671","title":"Qwen2 Technical Report","created":"2024-07-15","cited_by_count":2120},{"id":"2408.00118","title":"Gemma 2: Improving Open Language Models at a Practical Size","created":"2024-07-31","cited_by_count":1998},{"id":"1909.08053","title":"Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism","created":"2019-09-17","cited_by_count":1868},{"id":"2006.16668","title":"GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding","created":"2020-06-30","cited_by_count":1631},{"id":"1706.02677","title":"Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour","created":"2017-06-08","cited_by_count":1492},{"id":"1609.04836","title":"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima","created":"2016-09-15","cited_by_count":1352},{"id":"1910.02054","title":"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models","created":"2019-10-04","cited_by_count":1301},{"id":"1606.04838","title":"Optimization Methods for Large-Scale Machine Learning","created":"2016-06-15","cited_by_count":1150},{"id":"2403.08295","title":"Gemma: Open Models Based on Gemini Research and Technology","created":"2024-03-13","cited_by_count":916},{"id":"1904.00962","title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","created":"2019-04-01","cited_by_count":674},{"id":"2309.10305","title":"Baichuan 2: Open Large-scale Language Models","created":"2023-09-19","cited_by_count":670},{"id":"2304.11277","title":"PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel","created":"2023-04-21","cited_by_count":660},{"id":"2104.04473","title":"Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM","created":"2021-04-09","cited_by_count":632},{"id":"2401.02385","title":"TinyLlama: An Open-Source Small Language Model","created":"2024-01-04","cited_by_count":618},{"id":"2402.00838","title":"OLMo: Accelerating the Science of Language Models","created":"2024-02-01","cited_by_count":613},{"id":"2201.11990","title":"Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model","created":"2022-01-28","cited_by_count":559},{"id":"2101.06840","title":"ZeRO-Offload: Democratizing Billion-Scale Model Training","created":"2021-01-18","cited_by_count":341},{"id":"1711.00489","title":"Don't Decay the Learning Rate, Increase the Batch Size","created":"2017-11-01","cited_by_count":320},{"id":"2205.05198","title":"Reducing Activation Recomputation in Large Transformer Models","created":"2022-05-10","cited_by_count":302},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"1705.08741","title":"Train longer, generalize better: closing the generalization gap in large batch training of neural networks","created":"2017-05-24","cited_by_count":279},{"id":"1811.03600","title":"Measuring the Effects of Data Parallelism on Neural Network Training","created":"2018-11-08","cited_by_count":193},{"id":"2309.14322","title":"Small-scale proxies for large-scale Transformer training instabilities","created":"2023-09-25","cited_by_count":183},{"id":"2303.06296","title":"Stabilizing Transformer Training by Preventing Attention Entropy Collapse","created":"2023-03-11","cited_by_count":161},{"id":"2002.03329","title":"Better Theory for SGD in the Nonconvex World","created":"2020-02-09","cited_by_count":158},{"id":"1710.06451","title":"A Bayesian Perspective on Generalization and Stochastic Gradient Descent","created":"2017-10-17","cited_by_count":135},{"id":"2402.16788","title":"Why Transformers Need Adam: A Hessian Perspective","created":"2024-02-26","cited_by_count":124},{"id":"2401.08541","title":"Scalable Pre-training of Large Autoregressive Image Models","created":"2024-01-16","cited_by_count":115},{"id":"2304.13960","title":"Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be","created":"2023-04-27","cited_by_count":114},{"id":"2402.17834","title":"Stable LM 2 1.6B Technical Report","created":"2024-02-27","cited_by_count":90},{"id":"2410.06511","title":"TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training","created":"2024-10-09","cited_by_count":85},{"id":"2402.19449","title":"Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models","created":"2024-02-29","cited_by_count":80},{"id":"2410.21676","title":"How Does Critical Batch Size Scale in Pre-training?","created":"2024-10-29","cited_by_count":73},{"id":"2306.00204","title":"Toward Understanding Why Adam Converges Faster Than SGD for Transformers","created":"2023-05-31","cited_by_count":53},{"id":"2301.13442","title":"Scaling laws for single-agent reinforcement learning","created":"2023-01-31","cited_by_count":41},{"id":"2310.17998","title":"Closing the Gap Between the Upper Bound and the Lower Bound of Adam's Iteration Complexity","created":"2023-10-27","cited_by_count":38},{"id":"2108.06084","title":"The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models","created":"2021-08-13","cited_by_count":36},{"id":"2402.16819","title":"Nemotron-4 15B Technical Report","created":"2024-02-26","cited_by_count":36},{"id":"1612.05086","title":"Coupling Adaptive Batch Sizes with Learning Rates","created":"2016-12-15","cited_by_count":30},{"id":"1710.11258","title":"Adaptive Sampling Strategies for Stochastic Optimization","created":"2017-10-30","cited_by_count":30},{"id":"1712.02029","title":"AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks","created":"2017-12-06","cited_by_count":29},{"id":"2403.06563","title":"Unraveling the Mystery of Scaling Laws: Part I","created":"2024-03-11","cited_by_count":24},{"id":"1711.04325","title":"Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes","created":"2017-11-12","cited_by_count":22},{"id":"2007.05105","title":"AdaScale SGD: A User-Friendly Algorithm for Distributed Training","created":"2020-07-09","cited_by_count":16},{"id":"1610.05792","title":"Big Batch SGD: Automated Inference using Adaptive Batch Sizes","created":"2016-10-18","cited_by_count":13},{"id":"2309.10688","title":"On the different regimes of Stochastic Gradient Descent","created":"2023-09-19","cited_by_count":12},{"id":"2310.00576","title":"GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length","created":"2023-10-01","cited_by_count":11},{"id":"2402.11215","title":"AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods","created":"2024-02-17","cited_by_count":8},{"id":"1808.01371","title":"Large Scale Language Modeling: Converging on 40GB of Text in Four Hours","created":"2018-08-03","cited_by_count":4},{"id":"2109.12213","title":"Adaptive Sampling Quasi-Newton Methods for Zeroth-Order Stochastic Optimization","created":"2021-09-24","cited_by_count":4},{"id":"2406.13936","title":"Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods","created":"2024-06-20","cited_by_count":4},{"id":"2402.05264","title":"AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size","created":"2024-02-07","cited_by_count":3}],"next_cursor":null}