# Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi

arXiv: https://arxiv.org/abs/2407.09855 · HTML: https://mlsanity.com/paper/2407.09855/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 13 Jul 2024.

## Builds on (most cited first, 9 of 9)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages](https://mlsanity.com/paper/2210.09984/) (2022)
- [IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languages](https://mlsanity.com/paper/2203.05437/) (2022)
- [L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages](https://mlsanity.com/paper/2211.11418/) (2022)
- [Efficiently Adapting Pretrained Language Models To New Languages](https://mlsanity.com/paper/2311.05741/) (2023)
- [Assessing Translation capabilities of Large Language Models involving English and Indian Languages](https://mlsanity.com/paper/2311.09216/) (2023)
- [Synthetic data generation for Indic handwritten text recognition](https://mlsanity.com/paper/1804.06254/) (2018)

## Built on by (most cited first, 4 of 4)

- [Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research](https://mlsanity.com/paper/2412.04497/) (2024)
- [LIMBA: An Open-Source Framework for the Preservation and Valorization of Low-Resource Languages using Generative Models](https://mlsanity.com/paper/2411.13453/) (2024)
- [BYOL: Bring Your Own Language Into LLMs](https://mlsanity.com/paper/2601.10804/) (2026)
- [The Hrunting of AI: Where and How to Improve English Dialectal Fairness](https://mlsanity.com/paper/2603.15187/) (2026)

## Abstract (arXiv)

Large language models (LLMs) demonstrated transformative capabilities in many applications that require automatically generating responses based on human instruction. However, the major challenge for building LLMs, particularly in Indic languages, is the availability of high-quality data for building foundation LLMs. In this paper, we are proposing a large pre-train dataset in Hindi useful for the Indic language Hindi. We have collected the data span across several domains including major dialects in Hindi. The dataset contains 1.28 billion Hindi tokens. We have explained our pipeline including data collection, pre-processing, and availability for LLM pre-training. The proposed approach can be easily extended to other Indic and low-resource languages and will be available freely for LLM pre-training and LLM research purposes.