{"list":"other","total":84,"rows":[{"id":"1706.03762","title":"Attention Is All You Need","created":"2017-06-12","cited_by_count":60361},{"id":"1810.04805","title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","created":"2018-10-11","cited_by_count":40399},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"2303.08774","title":"GPT-4 Technical Report","created":"2023-03-15","cited_by_count":18396},{"id":"2407.21783","title":"The Llama 3 Herd of Models","created":"2024-07-31","cited_by_count":14715},{"id":"2307.09288","title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","created":"2023-07-18","cited_by_count":13028},{"id":"1503.02531","title":"Distilling the Knowledge in a Neural Network","created":"2015-03-09","cited_by_count":10324},{"id":"2505.09388","title":"Qwen3 Technical Report","created":"2025-05-14","cited_by_count":8820},{"id":"2304.08485","title":"Visual Instruction Tuning","created":"2023-04-17","cited_by_count":7869},{"id":"2001.08361","title":"Scaling Laws for Neural Language Models","created":"2020-01-23","cited_by_count":6586},{"id":"1803.05457","title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","created":"2018-03-14","cited_by_count":4886},{"id":"2012.12877","title":"Training data-efficient image transformers & distillation through attention","created":"2020-12-23","cited_by_count":4586},{"id":"1910.01108","title":"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter","created":"2019-10-02","cited_by_count":4307},{"id":"1905.07830","title":"HellaSwag: Can a Machine Really Finish Your Sentence?","created":"2019-05-19","cited_by_count":4100},{"id":"2205.01068","title":"OPT: Open Pre-trained Transformer Language Models","created":"2022-05-02","cited_by_count":3512},{"id":"1606.08415","title":"Gaussian Error Linear Units (GELUs)","created":"2016-06-27","cited_by_count":3502},{"id":"2203.15556","title":"Training Compute-Optimal Large Language Models","created":"2022-03-29","cited_by_count":3019},{"id":"1911.11641","title":"PIQA: Reasoning about Physical Commonsense in Natural Language","created":"2019-11-26","cited_by_count":2883},{"id":"2101.00027","title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling","created":"2020-12-31","cited_by_count":2411},{"id":"1905.10044","title":"BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions","created":"2019-05-24","cited_by_count":2183},{"id":"1809.02789","title":"Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering","created":"2018-09-08","cited_by_count":2164},{"id":"2002.05202","title":"GLU Variants Improve Transformer","created":"2020-02-12","cited_by_count":1804},{"id":"2304.01373","title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","created":"2023-04-03","cited_by_count":1786},{"id":"1910.07467","title":"Root Mean Square Layer Normalization","created":"2019-10-16","cited_by_count":1480},{"id":"2002.10957","title":"MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers","created":"2020-02-25","cited_by_count":1463},{"id":"2305.13245","title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints","created":"2023-05-22","cited_by_count":1346},{"id":"2112.11446","title":"Scaling Language Models: Methods, Analysis & Insights from Training Gopher","created":"2021-12-08","cited_by_count":1202},{"id":"1909.10351","title":"TinyBERT: Distilling BERT for Natural Language Understanding","created":"2019-09-23","cited_by_count":1167},{"id":"2210.02186","title":"TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis","created":"2022-10-05","cited_by_count":1058},{"id":"1606.06031","title":"The LAMBADA dataset: Word prediction requiring a broad discourse context","created":"2016-06-20","cited_by_count":992},{"id":"1702.03118","title":"Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning","created":"2017-02-10","cited_by_count":950},{"id":"1707.06209","title":"Crowdsourcing Multiple Choice Science Questions","created":"2017-07-19","cited_by_count":765},{"id":"2402.00838","title":"OLMo: Accelerating the Science of Language Models","created":"2024-02-01","cited_by_count":613},{"id":"1805.04770","title":"Born Again Neural Networks","created":"2018-05-12","cited_by_count":546},{"id":"2402.04333","title":"LESS: Selecting Influential Data for Targeted Instruction Tuning","created":"2024-02-06","cited_by_count":532},{"id":"2004.02984","title":"MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices","created":"2020-04-06","cited_by_count":482},{"id":"2312.09390","title":"Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision","created":"2023-12-14","cited_by_count":467},{"id":"2310.06694","title":"Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning","created":"2023-10-10","cited_by_count":415},{"id":"2305.10429","title":"DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining","created":"2023-05-17","cited_by_count":399},{"id":"2312.15685","title":"What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning","created":"2023-12-25","cited_by_count":364},{"id":"2502.02737","title":"SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model","created":"2025-02-04","cited_by_count":335},{"id":"2302.03169","title":"Data Selection for Language Models via Importance Resampling","created":"2023-02-06","cited_by_count":329},{"id":"2308.12032","title":"From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning","created":"2023-08-23","cited_by_count":307},{"id":"2205.05131","title":"UL2: Unifying Language Learning Paradigms","created":"2022-05-10","cited_by_count":295},{"id":"1711.04623","title":"Three Factors Influencing Minima in SGD","created":"2017-11-13","cited_by_count":273},{"id":"1511.05641","title":"Net2Net: Accelerating Learning via Knowledge Transfer","created":"2015-11-18","cited_by_count":250},{"id":"2407.13278","title":"Deep Time Series Models: A Comprehensive Survey and Benchmark","created":"2024-07-18","cited_by_count":197},{"id":"1903.12136","title":"Distilling Task-Specific Knowledge from BERT into Simple Neural Networks","created":"2019-03-28","cited_by_count":153},{"id":"2308.12284","title":"D4: Improving LLM Pretraining via Document De-Duplication and Diversification","created":"2023-08-23","cited_by_count":150},{"id":"1511.06807","title":"Adding Gradient Noise Improves Learning for Very Deep Networks","created":"2015-11-21","cited_by_count":145},{"id":"1511.06251","title":"Stochastic modified equations and adaptive stochastic gradient algorithms","created":"2015-11-19","cited_by_count":142},{"id":"2205.09329","title":"Dataset Pruning: Reducing Training Data by Examining Generalization Influence","created":"2022-05-19","cited_by_count":133},{"id":"2307.14430","title":"Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models","created":"2023-07-26","cited_by_count":121},{"id":"1707.06618","title":"Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimization","created":"2017-07-20","cited_by_count":97},{"id":"2210.01351","title":"Less is More: Task-aware Layer-wise Distillation for Language Model Compression","created":"2022-10-04","cited_by_count":93},{"id":"2006.08680","title":"Shape Matters: Understanding the Implicit Bias of the Noise Covariance","created":"2020-06-15","cited_by_count":88},{"id":"2303.00980","title":"Learning to Grow Pretrained Models for Efficient Transformer Training","created":"2023-03-02","cited_by_count":83},{"id":"2102.12470","title":"On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)","created":"2021-02-24","cited_by_count":73},{"id":"2110.07143","title":"bert2BERT: Towards Reusable Pretrained Language Models","created":"2021-10-14","cited_by_count":70},{"id":"2011.00593","title":"MixKD: Towards Efficient Distillation of Large-scale Language Models","created":"2020-11-01","cited_by_count":65},{"id":"2204.12997","title":"DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers","created":"2022-04-27","cited_by_count":65},{"id":"2106.04570","title":"BERT Learns to Teach: Knowledge Distillation with Meta Learning","created":"2021-06-08","cited_by_count":61},{"id":"2410.07524","title":"Upcycling Large Language Models into Mixture of Experts","created":"2024-10-10","cited_by_count":49},{"id":"1906.09069","title":"First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noise","created":"2019-06-21","cited_by_count":44},{"id":"2311.07052","title":"Towards the Law of Capacity Gap in Distilling Language Models","created":"2023-11-13","cited_by_count":44},{"id":"2405.15319","title":"Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training","created":"2024-05-24","cited_by_count":43},{"id":"2006.15815","title":"Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum","created":"2020-06-29","cited_by_count":40},{"id":"2105.13880","title":"Knowledge Inheritance for Pre-trained Language Models","created":"2021-05-28","cited_by_count":38},{"id":"2410.07064","title":"Data Selection via Optimal Control for Language Models","created":"2024-10-09","cited_by_count":38},{"id":"2502.09650","title":"Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples","created":"2025-02-11","cited_by_count":37},{"id":"2303.11098","title":"Understanding the Role of the Projector in Knowledge Distillation","created":"2023-03-20","cited_by_count":31},{"id":"2409.12903","title":"Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization","created":"2024-09-19","cited_by_count":29},{"id":"2011.06220","title":"Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting","created":"2020-11-12","cited_by_count":28},{"id":"2103.17182","title":"Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization","created":"2021-03-31","cited_by_count":28},{"id":"2305.10010","title":"AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression","created":"2023-05-17","cited_by_count":26},{"id":"2403.06213","title":"$V_kD:$ Improving Knowledge Distillation using Orthogonal Projections","created":"2024-03-10","cited_by_count":26},{"id":"2011.11152","title":"On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective","created":"2020-11-23","cited_by_count":25},{"id":"2310.07999","title":"LEMON: Lossless model expansion","created":"2023-10-12","cited_by_count":23},{"id":"2203.05180","title":"Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability","created":"2022-03-10","cited_by_count":21},{"id":"2503.13319","title":"MagicDistillation: Weak-to-Strong Video Distillation for Large-Scale Few-Step Synthesis","created":"2025-03-17","cited_by_count":19},{"id":"2410.18779","title":"A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs","created":"2024-10-24","cited_by_count":15},{"id":"2212.02083","title":"On the Overlooked Structure of Stochastic Gradients","created":"2022-12-05","cited_by_count":13},{"id":"2502.03009","title":"Scaling Laws for Upcycling Mixture-of-Experts Language Models","created":"2025-02-05","cited_by_count":13},{"id":"2305.18239","title":"A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models","created":"2023-05-26","cited_by_count":9}],"next_cursor":null}