{"list":"other","total":66,"rows":[{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2011.13456","title":"Score-Based Generative Modeling through Stochastic Differential Equations","created":"2020-11-26","cited_by_count":9122},{"id":"1608.03983","title":"SGDR: Stochastic Gradient Descent with Warm Restarts","created":"2016-08-13","cited_by_count":5045},{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019},{"id":"2203.05482","title":"Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time","created":"2022-03-10","cited_by_count":1396},{"id":"1803.05407","title":"Averaging Weights Leads to Wider Optima and Better Generalization","created":"2018-03-14","cited_by_count":1209},{"id":"2106.04560","title":"Scaling Vision Transformers","created":"2021-06-08","cited_by_count":999},{"id":"2302.00487","title":"A Comprehensive Survey of Continual Learning: Theory, Method and Application","created":"2023-01-31","cited_by_count":872},{"id":"2404.06395","title":"MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies","created":"2024-04-09","cited_by_count":723},{"id":"2402.00838","title":"OLMo: Accelerating the Science of Language Models","created":"2024-02-01","cited_by_count":613},{"id":"2312.02696","title":"Analyzing and Improving the Training Dynamics of Diffusion Models","created":"2023-12-05","cited_by_count":402},{"id":"1908.04742","title":"Online Continual Learning with Maximally Interfered Retrieval","created":"2019-08-11","cited_by_count":379},{"id":"2501.00656","title":"2 OLMo 2 Furious","created":"2024-12-31","cited_by_count":340},{"id":"2405.18392","title":"Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations","created":"2024-05-28","cited_by_count":170},{"id":"2405.15682","title":"The Road Less Scheduled","created":"2024-05-24","cited_by_count":163},{"id":"2209.15594","title":"Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability","created":"2022-09-30","cited_by_count":155},{"id":"2110.10832","title":"Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization","created":"2021-10-21","cited_by_count":144},{"id":"2308.04014","title":"Continual Pre-Training of Large Language Models: How to (re)warm your model?","created":"2023-08-08","cited_by_count":137},{"id":"2112.09153","title":"An Empirical Investigation of the Role of Pre-training in Lifelong Learning","created":"2021-12-16","cited_by_count":134},{"id":"2403.08763","title":"Simple and Scalable Strategies to Continually Pre-train Large Language Models","created":"2024-03-13","cited_by_count":127},{"id":"1306.2119","title":"Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)","created":"2013-06-10","cited_by_count":111},{"id":"2402.01092","title":"A Dynamical Model of Neural Scaling Laws","created":"2024-02-02","cited_by_count":111},{"id":"1812.05217","title":"Tight Analyses for Non-Smooth Stochastic Gradient Descent","created":"2018-12-13","cited_by_count":97},{"id":"2012.12631","title":"Efficient Continual Learning with Modular Networks and Task-Driven Priors","created":"2020-12-23","cited_by_count":88},{"id":"2205.09357","title":"Continual Pre-Training Mitigates Forgetting in Language and Vision","created":"2022-05-19","cited_by_count":83},{"id":"2410.05192","title":"Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective","created":"2024-10-07","cited_by_count":81},{"id":"1806.04498","title":"The Unusual Effectiveness of Averaging in GAN Training","created":"2018-06-12","cited_by_count":73},{"id":"2410.21676","title":"How Does Critical Batch Size Scale in Pre-training?","created":"2024-10-29","cited_by_count":73},{"id":"2103.12692","title":"Benign Overfitting of Constant-Stepsize SGD for Linear Regression","created":"2021-03-23","cited_by_count":71},{"id":"2411.18704","title":"Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits","created":"2024-11-27","cited_by_count":71},{"id":"1904.12838","title":"The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares","created":"2019-04-29","cited_by_count":68},{"id":"2405.15074","title":"4+3 Phases of Compute-Optimal Neural Scaling Laws","created":"2024-05-23","cited_by_count":68},{"id":"2406.08466","title":"Scaling Laws in Linear Regression: Compute, Parameters, and Data","created":"2024-06-12","cited_by_count":57},{"id":"2209.14981","title":"Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging","created":"2022-09-29","cited_by_count":54},{"id":"2306.07179","title":"Benchmarking Neural Network Training Algorithms","created":"2023-06-12","cited_by_count":52},{"id":"2405.00592","title":"Scaling and renormalization in high-dimensional regression","created":"2024-05-01","cited_by_count":50},{"id":"2406.09405","title":"Why Warmup the Learning Rate? Underlying Mechanisms and Improvements","created":"2024-06-13","cited_by_count":46},{"id":"2310.07831","title":"Optimal Linear Decay Learning Rate Schedules and Further Refinements","created":"2023-10-11","cited_by_count":43},{"id":"2409.19913","title":"Scaling Optimal LR Across Token Horizons","created":"2024-09-30","cited_by_count":40},{"id":"2110.06198","title":"Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regression","created":"2021-10-12","cited_by_count":38},{"id":"2306.03241","title":"Early Weight Averaging meets High Learning Rates for LLM Pre-training","created":"2023-06-05","cited_by_count":37},{"id":"2505.12082","title":"Model Merging in Pre-training of Large Language Models","created":"2025-05-17","cited_by_count":37},{"id":"2502.15938","title":"Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs","created":"2025-02-21","cited_by_count":35},{"id":"2501.18965","title":"The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training","created":"2025-01-31","cited_by_count":33},{"id":"2502.15015","title":"Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition","created":"2025-02-20","cited_by_count":32},{"id":"2408.11029","title":"Scaling Law with Learning Rate Annealing","created":"2024-08-20","cited_by_count":29},{"id":"2512.22382","title":"Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration","created":"2025-12-26","cited_by_count":27},{"id":"1710.09430","title":"A Markov Chain Theory Approach to Characterizing the Minimax Optimality of Stochastic Gradient Descent (for Least Squares)","created":"2017-10-25","cited_by_count":26},{"id":"2208.01857","title":"The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift","created":"2022-08-03","cited_by_count":26},{"id":"2412.01505","title":"Scaling Law for Language Models Training Considering Batch Size","created":"2024-12-02","cited_by_count":22},{"id":"2503.12811","title":"A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules","created":"2025-03-17","cited_by_count":22},{"id":"1412.6606","title":"Competing with the Empirical Risk Minimizer in a Single Pass","created":"2014-12-20","cited_by_count":20},{"id":"2106.09563","title":"On Anytime Learning at Macroscale","created":"2021-06-17","cited_by_count":20},{"id":"2509.19189","title":"Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules","created":"2025-09-23","cited_by_count":17},{"id":"2110.14109","title":"Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrums","created":"2021-10-27","cited_by_count":15},{"id":"2502.02431","title":"Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants","created":"2025-02-04","cited_by_count":10},{"id":"2402.09240","title":"Switch EMA: A Free Lunch for Better Flatness and Sharpness","created":"2024-02-14","cited_by_count":9},{"id":"2602.01480","title":"Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability","created":"2026-02-01","cited_by_count":9},{"id":"2409.09745","title":"The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization","created":"2024-09-15","cited_by_count":8},{"id":"2502.06761","title":"When, Where and Why to Average Weights?","created":"2025-02-10","cited_by_count":8},{"id":"2502.05074","title":"Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Models","created":"2025-02-07","cited_by_count":7},{"id":"2503.02844","title":"Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training","created":"2025-03-04","cited_by_count":7},{"id":"2510.00304","title":"Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity","created":"2025-09-30","cited_by_count":5},{"id":"2601.09000","title":"Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformers","created":"2026-01-13","cited_by_count":5},{"id":"2506.15535","title":"A Simplified Analysis of SGD for Linear Regression with Weight Averaging","created":"2025-06-18","cited_by_count":4}],"next_cursor":null}