mlsanityData to 7 Oct 2026

AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages

arXiv 2005.00085 ↗First posted 30 Apr 2020cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

We present the IndicNLP corpus, a large-scale, general-domain corpus containing 2.7 billion words for 10 Indian languages from two language families. We share pre-trained word embeddings trained on these corpora. We create news article category classification datasets for 9 languages to evaluate the embeddings. We show that the IndicNLP embeddings significantly outperform publicly available pre-trained embedding on multiple evaluation tasks. We hope that the availability of the corpus will accelerate Indic NLP research. The resources are available at https://github.com/ai4bharat-indicnlp/indicnlp_corpus.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 7; 2023: 6; 2024: 6; 2025: 8; 2026: 9.

43 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  3. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  4. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
  5. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  6. 2023Tamil-Llama: A New Tamil Language Model Based on Llama 2cited by 31
  7. 2023MasakhaNEWS: News Topic Classification for African languagescited by 30
  8. 2021Quantifying Gender Bias Towards Politicians in Cross-Lingual Language Modelscited by 15
  9. 2023L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERTcited by 15
  10. 2021A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Modelscited by 13
  11. 2025Tokenization Matters: Improving Zero-Shot NER for Indic Languagescited by 12
  12. 2022L3Cube-MahaSBERT and HindSBERT: Sentence BERT Models and Benchmarking BERT Sentence Representations for Hindi and Marathicited by 10
31 more
  1. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  2. 2024L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languagescited by 8
  3. 2021Evaluation of Deep Learning Models for Hostility Detection in Hindi Textcited by 7
  4. 2022Mitigating Gender Stereotypes in Hindi and Marathicited by 6
  5. 2024Standard Language Ideology in AI-Generated Languagecited by 5
  6. 2021Unsupervised Machine Translation On Dravidian Languagescited by 4
  7. 2025NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scriptscited by 4
  8. 2025SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methodscited by 4
  9. 2021Crosslingual Embeddings are Essential in UNMT for Distant Languages: An English to IndoAryan Case Studycited by 3
  10. 2021Cognition-aware Cognate Detectioncited by 3
  11. 2024HindiLLM: Large Language Model for Hindicited by 3
  12. 2022Cost-Effective Training in Low-Resource Neural Machine Translationcited by 2
  13. 2025BhashaKritika: Building Synthetic Pretraining Data at Scale for Indic Languagescited by 2
  14. 2024Krutrim LLM: A Novel Tokenization Strategy for Multilingual Indic Languages with Petabyte-Scale Data Processingcited by 1
  15. 2025BHRAM-IL: A Benchmark for Hallucination Recognition and Assessment in Multiple Indian Languagescited by 1
  16. 2021HinFlair: pre-trained contextual string embeddings for pos tagging and text classification in the Hindi languagecited by 0
  17. 2022Detecting Anchors' Opinion in Hinghlish News Deliverycited by 0
  18. 2022Part-of-Speech Tagging of Odia Language Using statistical and Deep Learning-Based Approachescited by 0
  19. 2023Cordyceps@LT-EDI: Patching Language-Specific Homophobia/Transphobia Classifiers with a Multilingual Understandingcited by 0
  20. 2024A Review of the Marathi Natural Language Processingcited by 0
  21. 2025Low-Resource Language Processing: An OCR-Driven Summarization and Translation Pipelinecited by 0
  22. 2025MahaParaphrase: A Marathi Paraphrase Detection Corpus and BERT-based Modelscited by 0
  23. 2026NepTam: A Nepali-Tamang Parallel Corpus and Baseline Machine Translation Experimentscited by 0
  24. 2026Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languagescited by 0
  25. 2026SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Palicited by 0
  26. 2026HiMed: Incentivizing Hindi Reasoning in Medical LLMscited by 0
  27. 2026BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysiscited by 0
  28. 2026BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruningcited by 0
  29. 2026IndicDetect: Evaluating Cross-Lingual LLM-Generated Text Detection for Hindi, Telugu, and Tamilcited by 0
  30. 2026Vectorizing Classical Tamil: Representation Learning for Verse-Commentary Pairscited by 0
  31. 2026Scaling Hindi Quantum Natural Language Processing through Automatic Pregroup Supertaggingcited by 0