{"list":"other","total":52,"rows":[{"id":"1810.04805","title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","created":"2018-10-11","cited_by_count":40399},{"id":"1412.6980","title":"Adam: A Method for Stochastic Optimization","created":"2014-12-22","cited_by_count":37240},{"id":"2010.11929","title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","created":"2020-10-22","cited_by_count":30186},{"id":"2303.08774","title":"GPT-4 Technical Report","created":"2023-03-15","cited_by_count":18396},{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"2307.09288","title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","created":"2023-07-18","cited_by_count":13028},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"2312.00752","title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","created":"2023-12-01","cited_by_count":4718},{"id":"1606.08415","title":"Gaussian Error Linear Units (GELUs)","created":"2016-06-27","cited_by_count":3502},{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019},{"id":"1711.05101","title":"Decoupled Weight Decay Regularization","created":"2017-11-14","cited_by_count":1831},{"id":"2002.05202","title":"GLU Variants Improve Transformer","created":"2020-02-12","cited_by_count":1804},{"id":"2010.01412","title":"Sharpness-Aware Minimization for Efficiently Improving Generalization","created":"2020-10-03","cited_by_count":1417},{"id":"1609.04836","title":"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima","created":"2016-09-15","cited_by_count":1352},{"id":"2305.13245","title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints","created":"2023-05-22","cited_by_count":1346},{"id":"2406.17557","title":"The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale","created":"2024-06-25","cited_by_count":1160},{"id":"2405.04434","title":"DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model","created":"2024-05-07","cited_by_count":1135},{"id":"2312.11805","title":"Gemini: A Family of Highly Capable Multimodal Models","created":"2023-12-19","cited_by_count":872},{"id":"1911.02150","title":"Fast Transformer Decoding: One Write-Head is All You Need","created":"2019-11-06","cited_by_count":717},{"id":"1712.00409","title":"Deep Learning Scaling is Predictable, Empirically","created":"2017-12-01","cited_by_count":582},{"id":"2405.04517","title":"xLSTM: Extended Long Short-Term Memory","created":"2024-05-07","cited_by_count":327},{"id":"2203.03466","title":"Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer","created":"2022-03-07","cited_by_count":288},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"2003.02218","title":"The large learning rate phase of deep learning: the catapult mechanism","created":"2020-03-04","cited_by_count":216},{"id":"2004.08249","title":"Understanding the Difficulty of Training Transformers","created":"2020-04-17","cited_by_count":210},{"id":"2309.14322","title":"Small-scale proxies for large-scale Transformer training instabilities","created":"2023-09-25","cited_by_count":183},{"id":"2405.15682","title":"The Road Less Scheduled","created":"2024-05-24","cited_by_count":163},{"id":"2209.15594","title":"Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability","created":"2022-09-30","cited_by_count":155},{"id":"1805.12076","title":"Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks","created":"2018-05-30","cited_by_count":142},{"id":"1710.06451","title":"A Bayesian Perspective on Generalization and Stochastic Gradient Descent","created":"2017-10-17","cited_by_count":135},{"id":"1810.12281","title":"Three Mechanisms of Weight Decay Regularization","created":"2018-10-29","cited_by_count":113},{"id":"2402.01092","title":"A Dynamical Model of Neural Scaling Laws","created":"2024-02-02","cited_by_count":111},{"id":"2207.14484","title":"Adaptive Gradient Methods at the Edge of Stability","created":"2022-07-29","cited_by_count":101},{"id":"2310.02244","title":"Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks","created":"2023-10-03","cited_by_count":97},{"id":"1709.01953","title":"Implicit Regularization in Deep Learning","created":"2017-09-06","cited_by_count":90},{"id":"2406.19146","title":"Resolving Discrepancies in Compute-Optimal Scaling of Language Models","created":"2024-06-27","cited_by_count":88},{"id":"2310.04415","title":"Why Do We Need Weight Decay in Modern Deep Learning?","created":"2023-10-06","cited_by_count":72},{"id":"2404.10102","title":"Chinchilla Scaling: A replication attempt","created":"2024-04-15","cited_by_count":72},{"id":"2405.15074","title":"4+3 Phases of Compute-Optimal Neural Scaling Laws","created":"2024-05-23","cited_by_count":68},{"id":"2309.16620","title":"Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit","created":"2023-09-28","cited_by_count":66},{"id":"2404.07839","title":"RecurrentGemma: Moving Past Transformers for Efficient Open Language Models","created":"2024-04-11","cited_by_count":64},{"id":"2406.08466","title":"Scaling Laws in Linear Regression: Compute, Parameters, and Data","created":"2024-06-12","cited_by_count":57},{"id":"2304.09871","title":"A Theory on Adam Instability in Large-Scale Machine Learning","created":"2023-04-19","cited_by_count":56},{"id":"2405.13698","title":"How to set AdamW's weight decay as you scale model and dataset size","created":"2024-05-22","cited_by_count":43},{"id":"2407.07972","title":"Deconstructing What Makes a Good Optimizer for Language Models","created":"2024-07-10","cited_by_count":41},{"id":"2210.04860","title":"Second-order regression models exhibit progressive sharpening to the edge of stability","created":"2022-10-10","cited_by_count":40},{"id":"2001.07301","title":"On the infinite width limit of neural networks with a standard parameterization","created":"2020-01-21","cited_by_count":38},{"id":"2407.17465","title":"u-$\\mu$P: The Unit-Scaled Maximal Update Parametrization","created":"2024-07-24","cited_by_count":29},{"id":"2306.08590","title":"Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning","created":"2023-06-14","cited_by_count":10},{"id":"2404.05728","title":"An Empirical Study of $\\mu$P Learning Rate Transfer","created":"2024-04-08","cited_by_count":10},{"id":"2408.02006","title":"LLaSA: Large Language and E-Commerce Shopping Assistant","created":"2024-08-04","cited_by_count":8},{"id":"2404.19719","title":"The lazy (NTK) and rich ($\\mu$P) regimes: a gentle tutorial","created":"2024-04-30","cited_by_count":2}],"next_cursor":null}