mlsanityData to 14 Oct 2026

Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation

arXiv 2505.00022 ↗First posted 24 Apr 2025cs.CL cs.AI cs.LGcited by 6

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Scaling data quantity is essential for large language models (LLMs), yet recent findings show that data quality can significantly boost performance and training efficiency. We introduce a German-language dataset curation pipeline that combines heuristic and model-based filtering techniques with synthetic data generation. We use our pipeline to create Aleph-Alpha-GermanWeb, a 628B-word German pre-training dataset composed of three subsets drawing from: (1) Common Crawl web data (organic subset; 78B words), (2) FineWeb2 (organic subset; 235B), and (3) synthetically-generated data conditioned on actual, organic web data (synthetic subset; 329B). We evaluate our dataset by pre-training both a 1B Llama-style model and an 8B tokeniser-free hierarchical autoregressive transformer (HAT) from scratch. A comparison on German-language benchmarks, including MMMLU, shows significant performance gains of Aleph-Alpha-GermanWeb over FineWeb2 alone. This advantage holds at the 8B scale even when FineWeb2 is enriched by human-curated high-quality data sources such as Wikipedia. Our findings support the growing body of evidence that model-based data curation and synthetic data generation can significantly enhance LLM pre-training datasets.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 32 of its 55 references that are in the map. First posted on arXiv 24 Apr 2025.

Builds on

32 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey14 Jun

Other references · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  12. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
19 more
  1. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  2. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  3. 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 170
  4. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  5. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  6. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  7. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  8. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 115
  9. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  10. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 43
  11. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 35
  12. 2024Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenizationcited by 32
  13. 2024Unraveling the Mystery of Scaling Laws: Part Icited by 24
  14. 2025Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Modelscited by 22
  15. 2023Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Productioncited by 21
  16. 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
  17. 2025LLMs on the Line: Data Determines Loss-to-Loss Scaling Lawscited by 16
  18. 2023CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Modelcited by 14
  19. 2024Problematic Tokens: Tokenizer Bias in Large Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026"UberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Datasetcited by 3
  2. 2026KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Reportcited by 1
  3. 2026A Family of LLMs Liberated from Static Vocabulariescited by 0
  4. 2026Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modelingcited by 0
  5. 2026From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inferencecited by 0
  6. 2026Fine PT-PT Web: A High-Quality 41 Billion Tokens Data Collection of the European Portuguese Webcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  2. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
  3. 2025Scaling Laws of Synthetic Data for Language Models
  4. 2025Blu-WERP (Web Extraction and Refinement Pipeline): A Scalable Pipeline for Preprocessing Large Language Model Datasets
  5. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  6. 2026Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods
  7. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
  8. 2024Rethinking KenLM: Good and Bad Model Ensembles for Efficient Text Quality Filtering in Large Web Corpora