# AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages

arXiv: https://arxiv.org/abs/2005.00085 · HTML: https://mlsanity.com/paper/2005.00085/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 30 of 44)

- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [Re-contextualizing Fairness in NLP: The Case of India](https://mlsanity.com/paper/2209.12226/) (2022)
- [Tamil-Llama: A New Tamil Language Model Based on Llama 2](https://mlsanity.com/paper/2311.05845/) (2023)
- [MasakhaNEWS: News Topic Classification for African languages](https://mlsanity.com/paper/2304.09972/) (2023)
- [Quantifying Gender Bias Towards Politicians in Cross-Lingual Language Models](https://mlsanity.com/paper/2104.07505/) (2021)
- [L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT](https://mlsanity.com/paper/2304.11434/) (2023)
- [A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models](https://mlsanity.com/paper/2107.03844/) (2021)
- [Tokenization Matters: Improving Zero-Shot NER for Indic Languages](https://mlsanity.com/paper/2504.16977/) (2025)
- [L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathi](https://mlsanity.com/paper/2211.11187/) (2022)
- [Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment](https://mlsanity.com/paper/2508.08424/) (2025)
- [L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages](https://mlsanity.com/paper/2401.02254/) (2024)
- [Evaluation of Deep Learning Models for Hostility Detection in Hindi Text](https://mlsanity.com/paper/2101.04144/) (2021)
- [Mitigating Gender Stereotypes in Hindi and Marathi](https://mlsanity.com/paper/2205.05901/) (2022)
- [Standard Language Ideology in AI-Generated Language](https://mlsanity.com/paper/2406.08726/) (2024)
- [Unsupervised Machine Translation On Dravidian Languages](https://mlsanity.com/paper/2103.15877/) (2021)
- [NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scripts](https://mlsanity.com/paper/2502.18148/) (2025)
- [SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methods](https://mlsanity.com/paper/2505.23714/) (2025)
- [Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Study](https://mlsanity.com/paper/2106.04995/) (2021)
- [Cognition-aware Cognate Detection](https://mlsanity.com/paper/2112.08087/) (2021)
- [HindiLLM: Large Language Model for Hindi](https://mlsanity.com/paper/2412.20357/) (2024)
- [Cost-Effective Training in Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2201.05700/) (2022)
- [Krutrim LLM: A Novel Tokenization Strategy for Multilingual Indic Languages with Petabyte-Scale Data Processing](https://mlsanity.com/paper/2407.12481/) (2024)
- [BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languages](https://mlsanity.com/paper/2511.10338/) (2025)
- [MahaParaphrase: A Marathi Paraphrase Detection Corpus and BERT-based Models](https://mlsanity.com/paper/2508.17444/) (2025)
- [BHRAM-IL: A Benchmark for Hallucination Recognition and Assessment in Multiple Indian Languages](https://mlsanity.com/paper/2512.01852/) (2025)
- [NepTam: A Nepali-Tamang Parallel Corpus and Baseline Machine Translation Experiments](https://mlsanity.com/paper/2603.14053/) (2026)
- [BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources](https://mlsanity.com/paper/2604.18423/) (2026)

## Abstract (arXiv)

We present the IndicNLP corpus, a large-scale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. We share pre-trained word embeddings trained on these corpora. We create news article category classification datasets for 9 languages to evaluate the embeddings. We show that the IndicNLP embeddings significantly outperform publicly available pre-trained embedding on multiple evaluation tasks. We hope that the availability of the corpus will accelerate Indic NLP research. The resources are available at https://github.com/ai4bharat-indicnlp/indicnlp_corpus.