mlsanityData to 14 Oct 2026

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

arXiv 2401.16380 ↗First posted 29 Jan 2024cs.CLcited by 161

Not on the map. Why?

Cited by 22 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the size of the model being trained. This is infeasible both because of the large compute costs and duration associated with pre-training, and the impending scarcity of high-quality data on the web. In this work, we propose Web Rephrase Augmented Pre-training (WRAP) that uses an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as "like Wikipedia" or in "question-answer format" to jointly pre-train LLMs on real and synthetic rephrases. First, we show that using WRAP on the C4 dataset, which is naturally noisy, speeds up pre-training by ~3x. At the same pre-training compute budget, it improves perplexity by more than 10% on average across different subsets of the Pile, and improves zero-shot question answer accuracy across 13 tasks by more than 2%. Second, we investigate the impact of the re-phrasing style on the performance of the model, offering insights into how the composition of the training data can impact the performance of LLMs in OOD settings. Our gains are attributed to the fact that re-phrased synthetic data has higher utility than just real data because it (i) incorporates style diversity that closely reflects downstream evaluation style, and (ii) has higher 'quality' than web-scraped data.

Key facts. Cited by 161 papers in the map (data through 14 Oct 2026). Builds on 54 of its 76 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

54 of 76 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2023Qwen Technical Reportcited by 3,344
38 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  12. 2023LIMA: Less Is More for Alignmentcited by 1,100
  13. 2023StarCoder: may the source be with you!cited by 799
  14. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  15. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  16. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  17. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  18. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  19. 2023Textbooks Are All You Needcited by 569
  20. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  21. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  22. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  23. 2023Scaling Data-Constrained Language Modelscited by 403
  24. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  25. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
  26. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  27. 2023Effective Data Augmentation With Diffusion Modelscited by 283
  28. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  29. 2023Self-Consuming Generative Models Go MADcited by 250
  30. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  31. 2023Self-Alignment with Instruction Backtranslationcited by 151
  32. 2023Online Speculative Decodingcited by 113
  33. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 92
  34. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  35. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  36. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  37. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 53
  38. 2023FinGPT: Large Generative Models for a Small Languagecited by 49

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 46; 2025: 58; 2026: 57.

161 papers.

Sort:

By area:

  1. 2025Kimi K2: Open Agentic Intelligencecited by 567
  2. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  3. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  4. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  5. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 119
  6. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 114
  7. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  8. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  9. 2024How to Train Data-Efficient LLMscited by 93
  10. 2025Self-Adapting Language Modelscited by 84
  11. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77
  12. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 76
38 more
  1. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  2. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 63
  3. 2024Synthetic continued pretrainingcited by 60
  4. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 51
  5. 2024Reverse Training to Nurse the Reversal Cursecited by 49
  6. 2025Diffusion Language Models are Super Data Learnerscited by 48
  7. 2025Scaling Laws of Synthetic Data for Language Modelscited by 46
  8. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 43
  9. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 43
  10. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  11. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  12. 2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 40
  13. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  14. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  15. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 32
  16. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  17. 2024ICONS: Influence Consensus for Vision-Language Data Selectioncited by 29
  18. 2025Pre-training under infinite computecited by 29
  19. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 26
  20. 2024How to Synthesize Text Data without Model Collapse?cited by 25
  21. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  22. 2024Language Models can Self-Lengthen to Generate Long Textscited by 24
  23. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 21
  24. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingcited by 21
  25. 2025A Survey on LLM Mid-Trainingcited by 21
  26. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 19
  27. 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
  28. 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
  29. 2024Better Alignment with Instruction Back-and-Forth Translationcited by 16
  30. 2024Towards Effective and Efficient Continual Pre-training of Large Language Modelscited by 15
  31. 2024CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentationcited by 14
  32. 2024MIND: Math Informed syNthetic Dialogues for Pretraining LLMscited by 14
  33. 2026Automated Optimization Modeling via a Localizable Error-Driven Perspectivecited by 14
  34. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Modelscited by 13
  35. 2024Self-calibration for Language Model Quantization and Pruningcited by 13
  36. 2024Zyda-2: a 5 Trillion Token High-Quality Datasetcited by 13
  37. 2025Group-Level Data Selection for Efficient Pretrainingcited by 13
  38. 2025Learning Facts at Scale with Active Readingcited by 13

50 most cited of 161 All 161 (JSON)