mlsanityData to 14 Oct 2026

Building a Large Japanese Web Corpus for Large Language Models

arXiv 2404.17733 ↗First posted 27 Apr 2024cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Open Japanese large language models (LLMs) have been trained on the Japanese portions of corpora such as CC-100, mC4, and OSCAR. However, these corpora were not created for the quality of Japanese texts. This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive (21 snapshots of approximately 63.4 billion pages crawled between 2020 and 2023). This corpus consists of approximately 312.1 billion characters (approximately 173 million pages), which is the largest of all available training corpora for Japanese LLMs, surpassing CC-100 (approximately 25.8 billion characters), mC4 (approximately 239.7 billion characters) and OSCAR 23.10 (approximately 74 billion characters). To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs and gained consistent (6.6-8.1 points) improvements on Japanese benchmark datasets. We also demonstrate that the improvement on Llama 2 13B brought from the presented corpus was the largest among those from other existing corpora.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 27 Apr 2024.

Builds on

33 of 48 references.

References · 33

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
21 more
  1. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  2. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  3. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  4. 2023A Watermark for Large Language Modelscited by 725
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  6. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  7. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  9. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  10. 2023Llemma: An Open Language Model For Mathematicscited by 388
  11. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  12. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  13. 2023How Language Model Hallucinations Can Snowballcited by 311
  14. 2023Large Language Model Unlearningcited by 302
  15. 2023Poisoning Language Models During Instruction Tuningcited by 269
  16. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  17. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  18. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 131
  19. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  20. 2022ClueWeb22: 10 Billion Web Documents with Visual and Semantic Informationcited by 20
  21. 2023The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 4.

11 papers.

  1. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 21
  2. 2025Kuwain 1.5B: An Arabic SLM via Language Injectioncited by 3
  3. 2025PLaMo 2 Technical Reportcited by 2
  4. 2025Model Correlation Detection via Random Selection Probingcited by 2
  5. 2026Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantizationcited by 2
  6. 2024Graph-Structured Trajectory Extraction from Traveloguescited by 1
  7. 2025Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Modelscited by 1
  8. 2026Suppressing Domain-Specific Hallucination in Construction LLMs: A Knowledge Graph Foundation for GraphRAG and QLoRA on River and Sediment Control Technical Standardscited by 1
  9. 2026Heterogeneous Graph Importance Scoring and Clustering with Automated LLM-based Interpretationcited by 1
  10. 2025Application of Contrastive Learning on ECG Data: Evaluating Performance in Japanese and Classification with Around 100 Labelscited by 0
  11. 2026Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Modelscited by 0