{"list":"other","total":83,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"1512.03385","title":"Deep Residual Learning for Image Recognition","created":"2015-12-10","cited_by_count":52096},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2307.09288","title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","created":"2023-07-18","cited_by_count":13028},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"1608.03983","title":"SGDR: Stochastic Gradient Descent with Warm Restarts","created":"2016-08-13","cited_by_count":5045},{"id":"1803.05457","title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","created":"2018-03-14","cited_by_count":4886},{"id":"2312.00752","title":"Mamba: Linear-Time Sequence Modeling with Selective State Spaces","created":"2023-12-01","cited_by_count":4718},{"id":"1905.07830","title":"HellaSwag: Can a Machine Really Finish Your Sentence?","created":"2019-05-19","cited_by_count":4100},{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019},{"id":"1911.11641","title":"PIQA: Reasoning about Physical Commonsense in Natural Language","created":"2019-11-26","cited_by_count":2883},{"id":"1905.00537","title":"SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems","created":"2019-05-02","cited_by_count":1729},{"id":"1706.02677","title":"Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour","created":"2017-06-08","cited_by_count":1492},{"id":"2106.04560","title":"Scaling Vision Transformers","created":"2021-06-08","cited_by_count":999},{"id":"1606.06031","title":"The LAMBADA dataset: Word prediction requiring a broad discourse context","created":"2016-06-20","cited_by_count":992},{"id":"2404.06395","title":"MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies","created":"2024-04-09","cited_by_count":723},{"id":"1603.06560","title":"Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization","created":"2016-03-21","cited_by_count":699},{"id":"2401.02954","title":"DeepSeek LLM: Scaling Open-Source Language Models with Longtermism","created":"2024-01-05","cited_by_count":642},{"id":"1506.01186","title":"Cyclical Learning Rates for Training Neural Networks","created":"2015-06-03","cited_by_count":629},{"id":"2402.00838","title":"OLMo: Accelerating the Science of Language Models","created":"2024-02-01","cited_by_count":613},{"id":"1712.00409","title":"Deep Learning Scaling is Predictable, Empirically","created":"2017-12-01","cited_by_count":582},{"id":"2010.14701","title":"Scaling Laws for Autoregressive Generative Modeling","created":"2020-10-28","cited_by_count":522},{"id":"2103.00065","title":"Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability","created":"2021-02-26","cited_by_count":427},{"id":"2305.16264","title":"Scaling Data-Constrained Language Models","created":"2023-05-25","cited_by_count":403},{"id":"2102.06701","title":"Explaining Neural Scaling Laws","created":"2021-02-12","cited_by_count":363},{"id":"2501.00656","title":"2 OLMo 2 Furious","created":"2024-12-31","cited_by_count":340},{"id":"1705.08741","title":"Train longer, generalize better: closing the generalization gap in large batch training of neural networks","created":"2017-05-24","cited_by_count":279},{"id":"2102.01293","title":"Scaling Laws for Transfer","created":"2021-02-02","cited_by_count":272},{"id":"2002.02561","title":"Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks","created":"2020-02-07","cited_by_count":200},{"id":"2403.16952","title":"Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance","created":"2024-03-25","cited_by_count":184},{"id":"2402.07871","title":"Scaling Laws for Fine-Grained Mixture of Experts","created":"2024-02-12","cited_by_count":173},{"id":"2405.18392","title":"Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations","created":"2024-05-28","cited_by_count":170},{"id":"2006.13198","title":"Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks","created":"2020-06-23","cited_by_count":166},{"id":"2405.15682","title":"The Road Less Scheduled","created":"2024-05-24","cited_by_count":163},{"id":"1907.04595","title":"Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks","created":"2019-07-10","cited_by_count":162},{"id":"2209.15594","title":"Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability","created":"2022-09-30","cited_by_count":155},{"id":"1511.06251","title":"Stochastic modified equations and adaptive stochastic gradient algorithms","created":"2015-11-19","cited_by_count":142},{"id":"2303.13506","title":"The Quantization Model of Neural Scaling","created":"2023-03-23","cited_by_count":138},{"id":"2301.03728","title":"Scaling Laws for Generative Mixed-Modal Language Models","created":"2023-01-10","cited_by_count":134},{"id":"2209.06640","title":"Revisiting Neural Scaling Laws in Language and Vision","created":"2022-09-13","cited_by_count":132},{"id":"2403.08763","title":"Simple and Scalable Strategies to Continually Pre-train Large Language Models","created":"2024-03-13","cited_by_count":127},{"id":"2310.05209","title":"Scaling Laws of RoPE-based Extrapolation","created":"2023-10-08","cited_by_count":125},{"id":"1910.07454","title":"An Exponential Learning Rate Schedule for Deep Learning","created":"2019-10-16","cited_by_count":121},{"id":"2210.16859","title":"A Solvable Model of Neural Scaling Laws","created":"2022-10-30","cited_by_count":116},{"id":"2402.01092","title":"A Dynamical Model of Neural Scaling Laws","created":"2024-02-02","cited_by_count":111},{"id":"2210.14891","title":"Broken Neural Scaling Laws","created":"2022-10-26","cited_by_count":109},{"id":"2205.10287","title":"On the SDEs and Scaling Rules for Adaptive Gradient Algorithms","created":"2022-05-20","cited_by_count":104},{"id":"2105.15004","title":"Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime","created":"2021-05-31","cited_by_count":96},{"id":"2403.15796","title":"Understanding Emergent Abilities of Language Models from the Loss Perspective","created":"2024-03-23","cited_by_count":94},{"id":"2203.06176","title":"More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize","created":"2022-03-11","cited_by_count":91},{"id":"2206.07085","title":"Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction","created":"2022-06-14","cited_by_count":86},{"id":"2102.04074","title":"Learning Curve Theory","created":"2021-02-08","cited_by_count":85},{"id":"1904.09679","title":"Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension","created":"2019-04-21","cited_by_count":81},{"id":"2212.14034","title":"Cramming: Training a Language Model on a Single GPU in One Day","created":"2022-12-28","cited_by_count":81},{"id":"2410.05192","title":"Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective","created":"2024-10-07","cited_by_count":81},{"id":"2404.07177","title":"Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic","created":"2024-04-10","cited_by_count":77},{"id":"2405.15074","title":"4+3 Phases of Compute-Optimal Neural Scaling Laws","created":"2024-05-23","cited_by_count":68},{"id":"2410.24206","title":"Understanding Optimization in Deep Learning with Central Flows","created":"2024-10-31","cited_by_count":64},{"id":"2406.08466","title":"Scaling Laws in Linear Regression: Compute, Parameters, and Data","created":"2024-06-12","cited_by_count":57},{"id":"2409.17858","title":"How Feature Learning Can Improve Neural Scaling Laws","created":"2024-09-26","cited_by_count":56},{"id":"2004.10802","title":"A Neural Scaling Law from the Dimension of the Data Manifold","created":"2020-04-22","cited_by_count":52},{"id":"2405.00592","title":"Scaling and renormalization in high-dimensional regression","created":"2024-05-01","cited_by_count":50},{"id":"2309.08520","title":"Scaling Laws for Sparsely-Connected Foundation Models","created":"2023-09-15","cited_by_count":43},{"id":"2310.07831","title":"Optimal Linear Decay Learning Rate Schedules and Further Refinements","created":"2023-10-11","cited_by_count":43},{"id":"2310.20447","title":"Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks","created":"2023-10-31","cited_by_count":37},{"id":"2502.15938","title":"Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs","created":"2025-02-21","cited_by_count":35},{"id":"2107.06608","title":"Continuous vs. Discrete Optimization of Deep Neural Networks","created":"2021-07-14","cited_by_count":34},{"id":"2009.06125","title":"A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms","created":"2020-09-14","cited_by_count":33},{"id":"2501.18965","title":"The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training","created":"2025-01-31","cited_by_count":33},{"id":"1909.09712","title":"Learning an Adaptive Learning Rate Schedule","created":"2019-09-20","cited_by_count":29},{"id":"2408.11029","title":"Scaling Law with Learning Rate Annealing","created":"2024-08-20","cited_by_count":29},{"id":"2408.13359","title":"Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler","created":"2024-08-23","cited_by_count":28},{"id":"2402.04376","title":"Scaling laws for learning with real and surrogate data","created":"2024-02-06","cited_by_count":27},{"id":"2405.14908","title":"BiMix: A Bivariate Data Mixing Law for Language Model Pretraining","created":"2024-05-23","cited_by_count":25},{"id":"2404.17563","title":"An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem","created":"2024-04-26","cited_by_count":22},{"id":"2302.00441","title":"Scaling Laws for Hyperparameter Optimization","created":"2023-02-01","cited_by_count":16},{"id":"2110.14109","title":"Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums","created":"2021-10-27","cited_by_count":15},{"id":"2411.12925","title":"Loss-to-Loss Prediction: Scaling Laws for All Datasets","created":"2024-11-19","cited_by_count":15},{"id":"2105.10762","title":"AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly","created":"2021-05-22","cited_by_count":14},{"id":"2409.04777","title":"Optimization Hyper-parameter Laws for Large Language Models","created":"2024-09-07","cited_by_count":7},{"id":"2111.15317","title":"AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop","created":"2021-11-30","cited_by_count":2}],"next_cursor":null}