{"list":"other","total":86,"rows":[{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2303.08774","title":"GPT-4 Technical Report","created":"2023-03-15","cited_by_count":18396},{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"2110.14168","title":"Training Verifiers to Solve Math Word Problems","created":"2021-10-27","cited_by_count":10109},{"id":"2103.03874","title":"Measuring Mathematical Problem Solving With the MATH Dataset","created":"2021-03-05","cited_by_count":5371},{"id":"1612.00796","title":"Overcoming catastrophic forgetting in neural networks","created":"2016-12-02","cited_by_count":4787},{"id":"2403.05530","title":"Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context","created":"2024-03-08","cited_by_count":3224},{"id":"2101.03961","title":"Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity","created":"2021-01-11","cited_by_count":3073},{"id":"2404.14219","title":"Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone","created":"2024-04-22","cited_by_count":1917},{"id":"2002.05202","title":"GLU Variants Improve Transformer","created":"2020-02-12","cited_by_count":1804},{"id":"1703.04730","title":"Understanding Black-box Predictions via Influence Functions","created":"2017-03-14","cited_by_count":1741},{"id":"2006.16668","title":"GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding","created":"2020-06-30","cited_by_count":1631},{"id":"2004.10964","title":"Don't Stop Pretraining: Adapt Language Models to Domains and Tasks","created":"2020-04-23","cited_by_count":1565},{"id":"2112.11446","title":"Scaling Language Models: Methods, Analysis & Insights from Training Gopher","created":"2021-12-08","cited_by_count":1202},{"id":"2108.12409","title":"Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation","created":"2021-08-27","cited_by_count":1103},{"id":"2403.08295","title":"Gemma: Open Models Based on Gemini Research and Technology","created":"2024-03-13","cited_by_count":916},{"id":"1804.04235","title":"Adafactor: Adaptive Learning Rates with Sublinear Memory Cost","created":"2018-04-11","cited_by_count":905},{"id":"2202.07646","title":"Quantifying Memorization Across Neural Language Models","created":"2022-02-15","cited_by_count":763},{"id":"2404.06395","title":"MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies","created":"2024-04-09","cited_by_count":723},{"id":"2401.02954","title":"DeepSeek LLM: Scaling Open-Source Language Models with Longtermism","created":"2024-01-05","cited_by_count":642},{"id":"2201.02177","title":"Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets","created":"2022-01-06","cited_by_count":629},{"id":"1308.0850","title":"Generating Sequences With Recurrent Neural Networks","created":"2013-08-04","cited_by_count":554},{"id":"2402.04333","title":"LESS: Selecting Influential Data for Targeted Instruction Tuning","created":"2024-02-06","cited_by_count":532},{"id":"2008.03703","title":"What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation","created":"2020-08-09","cited_by_count":497},{"id":"1802.09568","title":"Shampoo: Preconditioned Stochastic Tensor Optimization","created":"2018-02-26","cited_by_count":426},{"id":"2406.11794","title":"DataComp-LM: In search of the next generation of training sets for language models","created":"2024-06-17","cited_by_count":398},{"id":"2501.00656","title":"2 OLMo 2 Furious","created":"2024-12-31","cited_by_count":340},{"id":"2308.03296","title":"Studying Large Language Model Generalization with Influence Functions","created":"2023-08-07","cited_by_count":326},{"id":"1812.06162","title":"An Empirical Model of Large-Batch Training","created":"2018-12-14","cited_by_count":281},{"id":"2305.14342","title":"Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training","created":"2023-05-23","cited_by_count":265},{"id":"2106.04426","title":"Hash Layers For Large Sparse Models","created":"2021-06-08","cited_by_count":237},{"id":"2206.07137","title":"Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt","created":"2022-06-14","cited_by_count":236},{"id":"1811.03600","title":"Measuring the Effects of Data Parallelism on Neural Network Training","created":"2018-11-08","cited_by_count":193},{"id":"2409.11321","title":"SOAP: Improving and Stabilizing Shampoo using Adam","created":"2024-09-17","cited_by_count":192},{"id":"2403.16952","title":"Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance","created":"2024-03-25","cited_by_count":184},{"id":"2407.01492","title":"RegMix: Data Mixture as Regression for Language Model Pre-training","created":"2024-07-01","cited_by_count":177},{"id":"2303.01486","title":"Understanding plasticity in neural networks","created":"2023-03-02","cited_by_count":174},{"id":"2405.18392","title":"Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations","created":"2024-05-28","cited_by_count":170},{"id":"2109.08668","title":"Primer: Searching for Efficient Transformers for Language Modeling","created":"2021-09-17","cited_by_count":164},{"id":"2205.10487","title":"Scaling Laws and Interpretability of Learning from Repeated Data","created":"2022-05-21","cited_by_count":152},{"id":"2403.08763","title":"Simple and Scalable Strategies to Continually Pre-train Large Language Models","created":"2024-03-13","cited_by_count":127},{"id":"2310.00902","title":"DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models","created":"2023-10-02","cited_by_count":123},{"id":"1907.04164","title":"Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model","created":"2019-07-09","cited_by_count":117},{"id":"2506.20512","title":"OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling","created":"2025-06-25","cited_by_count":113},{"id":"2304.03208","title":"Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster","created":"2023-04-06","cited_by_count":109},{"id":"2411.04330","title":"Scaling Laws for Precision","created":"2024-11-07","cited_by_count":95},{"id":"2404.15146","title":"Rethinking LLM Memorization through the Lens of Adversarial Compression","created":"2024-04-23","cited_by_count":94},{"id":"2405.13954","title":"What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions","created":"2024-05-22","cited_by_count":94},{"id":"2402.09668","title":"How to Train Data-Efficient LLMs","created":"2024-02-15","cited_by_count":93},{"id":"2406.19146","title":"Resolving Discrepancies in Compute-Optimal Scaling of Language Models","created":"2024-06-27","cited_by_count":88},{"id":"2505.01618","title":"Don't be lazy: CompleteP enables compute-efficient deep transformers","created":"2025-05-02","cited_by_count":78},{"id":"2410.21676","title":"How Does Critical Batch Size Scale in Pre-training?","created":"2024-10-29","cited_by_count":73},{"id":"2404.10102","title":"Chinchilla Scaling: A replication attempt","created":"2024-04-15","cited_by_count":72},{"id":"2407.07263","title":"Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models","created":"2024-07-09","cited_by_count":72},{"id":"2203.06311","title":"ELLE: Efficient Lifelong Pre-training for Emerging Data","created":"2022-03-12","cited_by_count":70},{"id":"2405.19327","title":"MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series","created":"2024-05-29","cited_by_count":67},{"id":"2505.24832","title":"How much do language models memorize?","created":"2025-05-30","cited_by_count":61},{"id":"2406.06563","title":"Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models","created":"2024-06-03","cited_by_count":59},{"id":"2503.19206","title":"Overtrained Language Models Are Harder to Fine-Tune","created":"2025-03-24","cited_by_count":58},{"id":"2404.07413","title":"JetMoE: Reaching Llama2 Performance with 0.1M Dollars","created":"2024-04-11","cited_by_count":57},{"id":"2507.22448","title":"Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance","created":"2025-07-30","cited_by_count":57},{"id":"2409.03137","title":"The AdEMAMix Optimizer: Better, Faster, Older","created":"2024-09-05","cited_by_count":54},{"id":"2406.03476","title":"Does your data spark joy? Performance gains from domain upsampling at the end of training","created":"2024-06-05","cited_by_count":52},{"id":"2410.05355","title":"Falcon Mamba: The First Competitive Attention-free 7B Language Model","created":"2024-10-07","cited_by_count":46},{"id":"2405.13698","title":"How to set AdamW's weight decay as you scale model and dataset size","created":"2024-05-22","cited_by_count":43},{"id":"2505.13738","title":"Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training","created":"2025-05-19","cited_by_count":43},{"id":"2305.18411","title":"Feature-Learning Networks Are Consistent Across Widths At Realistic Scales","created":"2023-05-28","cited_by_count":41},{"id":"2412.09538","title":"Capturing the Temporal Dependence of Training Data Influence","created":"2024-12-12","cited_by_count":35},{"id":"2502.15938","title":"Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs","created":"2025-02-21","cited_by_count":35},{"id":"2407.07890","title":"Training on the Test Task Confounds Evaluation and Emergence","created":"2024-07-10","cited_by_count":34},{"id":"2501.18965","title":"The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training","created":"2025-01-31","cited_by_count":33},{"id":"2408.11029","title":"Scaling Law with Learning Rate Annealing","created":"2024-08-20","cited_by_count":29},{"id":"2412.15285","title":"Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining","created":"2024-12-18","cited_by_count":26},{"id":"2402.17457","title":"Super Consistency of Neural Network Landscapes and Learning Rate Transfer","created":"2024-02-27","cited_by_count":23},{"id":"2505.23971","title":"Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training","created":"2025-05-29","cited_by_count":23},{"id":"2311.02076","title":"Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos","created":"2023-11-03","cited_by_count":22},{"id":"2410.23922","title":"Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training","created":"2024-10-31","cited_by_count":22},{"id":"2507.02119","title":"Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks","created":"2025-07-02","cited_by_count":22},{"id":"2410.19034","title":"Mixture of Parrots: Experts improve memorization more than reasoning","created":"2024-10-24","cited_by_count":21},{"id":"2406.04327","title":"Causal Estimation of Memorisation Profiles","created":"2024-06-06","cited_by_count":20},{"id":"2408.08310","title":"ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws","created":"2024-08-15","cited_by_count":15},{"id":"2504.07866","title":"Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs","created":"2025-04-10","cited_by_count":13},{"id":"2412.01253","title":"Yi-Lightning Technical Report","created":"2024-12-02","cited_by_count":12},{"id":"2410.05581","title":"Adaptation Odyssey in LLMs: Why Does Additional Pretraining Sometimes Fail to Improve?","created":"2024-10-08","cited_by_count":8},{"id":"2506.16884","title":"The Importance of Being Lazy: Scaling Limits of Continual Learning","created":"2025-06-20","cited_by_count":6},{"id":"2505.07293","title":"AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection","created":"2025-05-12","cited_by_count":5}],"next_cursor":null}