{"list":"other","total":70,"rows":[{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"1910.10683","title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","created":"2019-10-23","cited_by_count":14599},{"id":"2302.13971","title":"LLaMA: Open and Efficient Foundation Language Models","created":"2023-02-27","cited_by_count":14132},{"id":"1503.02531","title":"Distilling the Knowledge in a Neural Network","created":"2015-03-09","cited_by_count":10324},{"id":"2107.03374","title":"Evaluating Large Language Models Trained on Code","created":"2021-07-07","cited_by_count":8320},{"id":"2009.03300","title":"Measuring Massive Multitask Language Understanding","created":"2020-09-07","cited_by_count":7963},{"id":"1803.05457","title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","created":"2018-03-14","cited_by_count":4886},{"id":"2104.09864","title":"RoFormer: Enhanced Transformer with Rotary Position Embedding","created":"2021-04-20","cited_by_count":4703},{"id":"1905.07830","title":"HellaSwag: Can a Machine Really Finish Your Sentence?","created":"2019-05-19","cited_by_count":4100},{"id":"1911.11641","title":"PIQA: Reasoning about Physical Commonsense in Natural Language","created":"2019-11-26","cited_by_count":2883},{"id":"2101.00027","title":"The Pile: An 800GB Dataset of Diverse Text for Language Modeling","created":"2020-12-31","cited_by_count":2411},{"id":"1905.10044","title":"BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions","created":"2019-05-24","cited_by_count":2183},{"id":"1809.02789","title":"Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering","created":"2018-09-08","cited_by_count":2164},{"id":"1811.00937","title":"CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge","created":"2018-11-02","cited_by_count":1822},{"id":"2002.05202","title":"GLU Variants Improve Transformer","created":"2020-02-12","cited_by_count":1804},{"id":"2112.11446","title":"Scaling Language Models: Methods, Analysis & Insights from Training Gopher","created":"2021-12-08","cited_by_count":1202},{"id":"2406.17557","title":"The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale","created":"2024-06-25","cited_by_count":1160},{"id":"2312.07104","title":"SGLang: Efficient Execution of Structured Language Model Programs","created":"2023-12-12","cited_by_count":1141},{"id":"1506.06724","title":"Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books","created":"2015-06-22","cited_by_count":1043},{"id":"1607.01759","title":"Bag of Tricks for Efficient Text Classification","created":"2016-07-06","cited_by_count":876},{"id":"2305.14233","title":"Enhancing Chat Language Models by Scaling High-quality Instructional Conversations","created":"2023-05-23","cited_by_count":839},{"id":"2308.03281","title":"Towards General Text Embeddings with Multi-stage Contrastive Learning","created":"2023-08-07","cited_by_count":723},{"id":"2107.06499","title":"Deduplicating Training Data Makes Language Models Better","created":"2021-07-14","cited_by_count":714},{"id":"2204.06745","title":"GPT-NeoX-20B: An Open-Source Autoregressive Language Model","created":"2022-04-14","cited_by_count":713},{"id":"1911.00359","title":"CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data","created":"2019-11-01","cited_by_count":548},{"id":"2402.00159","title":"Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research","created":"2024-01-31","cited_by_count":483},{"id":"2305.16264","title":"Scaling Data-Constrained Language Models","created":"2023-05-25","cited_by_count":403},{"id":"2305.10429","title":"DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining","created":"2023-05-17","cited_by_count":399},{"id":"2406.11794","title":"DataComp-LM: In search of the next generation of training sets for language models","created":"2024-06-17","cited_by_count":398},{"id":"2501.00656","title":"2 OLMo 2 Furious","created":"2024-12-31","cited_by_count":340},{"id":"2302.03169","title":"Data Selection for Language Models via Importance Resampling","created":"2023-02-06","cited_by_count":329},{"id":"2303.09540","title":"SemDeDup: Data-efficient learning at web-scale through semantic deduplication","created":"2023-03-16","cited_by_count":303},{"id":"2403.16952","title":"Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance","created":"2024-03-25","cited_by_count":184},{"id":"2305.13169","title":"A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity","created":"2023-05-22","cited_by_count":182},{"id":"2407.01492","title":"RegMix: Data Mixture as Regression for Language Model Pre-training","created":"2024-07-01","cited_by_count":177},{"id":"2401.16380","title":"Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling","created":"2024-01-29","cited_by_count":161},{"id":"2402.09739","title":"QuRating: Selecting High-Quality Data for Training Language Models","created":"2024-02-15","cited_by_count":157},{"id":"2308.12284","title":"D4: Improving LLM Pretraining via Document De-Duplication and Diversification","created":"2023-08-23","cited_by_count":150},{"id":"2310.20707","title":"What's In My Big Data?","created":"2023-10-31","cited_by_count":129},{"id":"2307.14430","title":"Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models","created":"2023-07-26","cited_by_count":121},{"id":"2406.08446","title":"OLMES: A Standard for Language Model Evaluations","created":"2024-06-12","cited_by_count":106},{"id":"2401.12926","title":"DsDm: Model-Aware Dataset Selection with Datamodels","created":"2024-01-23","cited_by_count":105},{"id":"2406.06046","title":"MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models","created":"2024-06-10","cited_by_count":100},{"id":"2402.09668","title":"How to Train Data-Efficient LLMs","created":"2024-02-15","cited_by_count":93},{"id":"2310.15393","title":"DoGE: Domain Reweighting with Generalization Estimation","created":"2023-10-23","cited_by_count":90},{"id":"2405.20541","title":"Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models","created":"2024-05-30","cited_by_count":84},{"id":"2309.16298","title":"At Which Training Stage Does Code Data Help LLMs Reasoning?","created":"2023-09-28","cited_by_count":82},{"id":"2312.02406","title":"Efficient Online Data Mixing For Language Model Pre-Training","created":"2023-12-05","cited_by_count":77},{"id":"2412.13678","title":"Clio: Privacy-Preserving Insights into Real-World AI Use","created":"2024-12-18","cited_by_count":72},{"id":"2305.14871","title":"ClusterLLM: Large Language Models as a Guide for Text Clustering","created":"2023-05-24","cited_by_count":70},{"id":"2303.14177","title":"Scaling Expert Language Models with Unsupervised Domain Discovery","created":"2023-03-24","cited_by_count":60},{"id":"2305.13749","title":"Goal-Driven Explainable Clustering via Language Descriptions","created":"2023-05-23","cited_by_count":58},{"id":"2201.12323","title":"Describing Differences between Text Distributions with Natural Language","created":"2022-01-28","cited_by_count":54},{"id":"2405.15613","title":"Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach","created":"2024-05-24","cited_by_count":50},{"id":"2410.08527","title":"Scaling Laws for Predicting Downstream Performance in LLMs","created":"2024-10-11","cited_by_count":48},{"id":"2408.10914","title":"To Code, or Not To Code? Exploring Impact of Code in Pre-training","created":"2024-08-20","cited_by_count":47},{"id":"2410.11820","title":"Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws","created":"2024-10-15","cited_by_count":47},{"id":"2407.14933","title":"Consent in Crisis: The Rapid Decline of the AI Data Commons","created":"2024-07-20","cited_by_count":46},{"id":"2409.05816","title":"Improving Pretraining Data Using Perplexity Correlations","created":"2024-09-09","cited_by_count":46},{"id":"2411.12580","title":"Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models","created":"2024-11-19","cited_by_count":41},{"id":"2412.04403","title":"Establishing Task Scaling Laws via Compute-Efficient Model Ladders","created":"2024-12-05","cited_by_count":35},{"id":"2409.16986","title":"Harnessing Diversity for Important Data Selection in Pretraining Large Language Models","created":"2024-09-25","cited_by_count":34},{"id":"2411.05735","title":"Aioli: A Unified Optimization Framework for Language Model Data Mixing","created":"2024-11-08","cited_by_count":34},{"id":"2409.04556","title":"How Does Code Pretraining Affect Language Model Task Performance?","created":"2024-09-06","cited_by_count":28},{"id":"2401.06408","title":"AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters","created":"2024-01-12","cited_by_count":22},{"id":"2501.01956","title":"Metadata Conditioning Accelerates Language Model Pre-training","created":"2025-01-03","cited_by_count":22},{"id":"2010.12813","title":"Constructing Taxonomies from Pretrained Language Models","created":"2020-10-24","cited_by_count":21},{"id":"2406.10670","title":"CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training","created":"2024-06-15","cited_by_count":20},{"id":"2501.11747","title":"Optimizing Pretraining Data Mixtures with LLM-Estimated Utility","created":"2025-01-20","cited_by_count":15},{"id":"2402.13623","title":"FLAME: Self-Supervised Low-Resource Taxonomy Expansion using Large Language Models","created":"2024-02-21","cited_by_count":4}],"next_cursor":null}