mlsanityData to 14 Oct 2026

BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages

arXiv 2511.10338 ↗First posted 13 Nov 2025cs.CL cs.AIcited by 2

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

In the context of pretraining of Large Language Models (LLMs), synthetic data has emerged as an alternative for generating high-quality pretraining data at scale. This is particularly beneficial in low-resource language settings where the benefits of recent LLMs have been unevenly distributed across languages. In this work, we present a systematic study on the generation and evaluation of synthetic multilingual pretraining data for Indic languages, where we construct a large-scale synthetic dataset BhashaKritika, comprising 540B tokens using 5 different techniques for 10 languages. We explore the impact of grounding generation in documents, personas, and topics. We analyze how language choice, both in the prompt instructions and document grounding, affects data quality, and we compare translations of English content with native generation in Indic languages. To support scalable and language-sensitive evaluation, we introduce a modular quality evaluation pipeline that integrates script and language detection, metadata consistency checks, n-gram repetition analysis, and perplexity-based filtering using KenLM models. Our framework enables robust quality control across diverse scripts and linguistic contexts. Empirical results through model runs reveal key trade-offs in generation strategies and highlight best practices for constructing effective multilingual corpora.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 48 of its 69 references that are in the map. First posted on arXiv 13 Nov 2025.

Builds on

48 of 69 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Scaling Laws of Synthetic Data for Language Models25 Mar
  2. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code18 Mar
  3. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personas28 Jun
  4. 2024Best Practices and Lessons Learned on Synthetic Data11 Apr

Other references · 44

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
32 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  5. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  6. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  9. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  10. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  11. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  12. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  13. 2016FastText.zip: Compressing text classification modelscited by 335
  14. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  15. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  16. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  17. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
  18. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  19. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  20. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
  21. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 102
  22. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 58
  23. 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
  24. 2020AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languagescited by 44
  25. 2023Tamil-Llama: A New Tamil Language Model Based on Llama 2cited by 32
  26. 2024Airavata: Introducing Hindi Instruction-tuned LLMcited by 32
  27. 2023Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Setcited by 12
  28. 2024Scaling Parameter-Constrained Language Models with Quality Datacited by 12
  29. 2024SUTRA: Scalable Multilingual Language Model Architecturecited by 11
  30. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  31. 2024Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progresscited by 7
  32. 2025Language Specific Knowledge: Do Models Know Better in X than in English?cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Code Mixologist : A Practitioner's Guide to Building Code-Mixed LLMscited by 0
  2. 2026Rethinking Indic AI from a Lens of Cultural Heritage Preservationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  2. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  3. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
  4. 2026A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books
  5. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Models
  6. 2024Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs
  7. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  8. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data