mlsanityData to 14 Oct 2026

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

arXiv 2306.01116 ↗First posted 1 Jun 2023cs.CL cs.AIcited by 763

Not on the map. Why?

Cited by 51 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.

Key facts. Cited by 763 papers in the map (data through 14 Oct 2026). Builds on 59 of its 95 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

59 of 95 references.

References · 59

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  11. 2018Datasheets for Datasetscited by 1,386
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  13. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  14. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  15. 2018Model Cards for Model Reportingcited by 1,080
  16. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  17. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  18. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  19. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  20. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  21. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  22. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  23. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  24. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  25. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  26. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  27. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  28. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  29. 2018Learning Word Vectors for 157 Languagescited by 280
  30. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  31. 2018A Simple Method for Commonsense Reasoningcited by 223
  32. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  33. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  34. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  35. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  36. 2021Challenges in Detoxifying Language Modelscited by 169
  37. 2022Compute Trends Across Three Eras of Machine Learningcited by 163
  38. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 210; 2024: 345; 2025: 137; 2026: 71.

763 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  7. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  9. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  10. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  11. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
38 more
  1. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  3. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  4. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  6. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  7. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  8. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  9. 2023The Falcon Series of Open Language Modelscited by 535
  10. 2024Large Language Models: A Surveycited by 525
  11. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  12. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  13. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  14. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  15. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  16. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  17. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  18. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  19. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  20. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  21. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  22. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  23. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  24. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  25. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  26. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  27. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  28. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 287
  29. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  30. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  31. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  32. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  33. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  34. 2023Challenges and Applications of Large Language Modelscited by 254
  35. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  36. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  37. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  38. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245

50 most cited of 763 All 763 (JSON)