mlsanityData to 7 Oct 2026

Decoding the Diversity: A Review of the Indic AI Research Landscape

arXiv 2406.09559 ↗First posted 13 Jun 2024cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

This review paper provides a comprehensive overview of large language model (LLM) research directions within Indic languages. Indic languages are those spoken in the Indian subcontinent, including India, Pakistan, Bangladesh, Sri Lanka, Nepal, and Bhutan, among others. These languages have a rich cultural and linguistic heritage and are spoken by over 1.5 billion people worldwide. With the tremendous market potential and growing demand for natural language processing (NLP) based applications in diverse languages, generative applications for Indic languages pose unique challenges and opportunities for research. Our paper deep dives into the recent advancements in Indic generative modeling, contributing with a taxonomy of research directions, tabulating 84 recent publications. Research directions surveyed in this paper include LLM development, fine-tuning existing LLMs, development of corpora, benchmarking and evaluation, as well as publications around specific techniques, tools, and applications. We found that researchers across the publications emphasize the challenges associated with limited data availability, lack of standardization, and the peculiar linguistic complexities of Indic languages. This work aims to serve as a valuable resource for researchers and practitioners working in the field of NLP, particularly those focused on Indic languages, and contributes to the development of more accurate and efficient LLM applications for these languages.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 55 of its 86 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

55 of 86 references.

References · 55

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  5. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  6. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  7. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  8. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  9. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  10. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  11. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  12. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 167
38 more
  1. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
  2. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  3. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  4. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  5. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  6. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  7. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  8. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  9. 2023Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Promptscited by 45
  10. 2022IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languagescited by 42
  11. 2020PMIndia -- A Collection of Parallel Corpora of Languages of Indiacited by 40
  12. 2022BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Banglacited by 40
  13. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  14. 2023Tamil-Llama: A New Tamil Language Model Based on Llama 2cited by 31
  15. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  16. 2022IndicXNLI: Evaluating Multilingual Inference for Indian Languagescited by 26
  17. 2024Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applicationscited by 25
  18. 2020iNLTK: Natural Language Toolkit for Indic Languagescited by 24
  19. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
  20. 2023BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLPcited by 23
  21. 2020Indic-Transformers: An Analysis of Transformer Language Models for Indian Languagescited by 22
  22. 2024Cross-lingual Editing in Multilingual Language Modelscited by 18
  23. 2022HateCheckHIn: Evaluating Hindi Hate Speech Detection Modelscited by 15
  24. 2023Large-scale Language Model Rescoring on Long-form Datacited by 14
  25. 2024adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgroundscited by 14
  26. 2023Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Setcited by 12
  27. 2024Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?cited by 12
  28. 2023Can LLMs be Good Financial Advisors?: An Initial Study in Personal Decision Making for Optimized Outcomescited by 10
  29. 2023Lost in Translation, Found in Spans: Identifying Claims in Multilingual Social Mediacited by 8
  30. 2023Assessing Translation capabilities of Large Language Models involving English and Indian Languagescited by 8
  31. 2023LLM-RM at SemEval-2023 Task 2: Multilingual Complex NER using XLM-RoBERTacited by 7
  32. 2020BERT Based Multilingual Machine Comprehension in English and Hindicited by 5
  33. 2024BanglaAutoKG: Automatic Bangla Knowledge Graph Construction with Semantic Neural Graph Filteringcited by 5
  34. 2021Language Identification of Hindi-English tweets using code-mixed BERTcited by 4
  35. 2023Mixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindicited by 4
  36. 2024IndiText Boost: Text Augmentation for Low Resource India Languagescited by 4
  37. 2023Intent Detection and Slot Filling for Home Assistants: Dataset and Analysis for Bangla and Sylheticited by 3
  38. 2023BanMANI: A Dataset to Identify Manipulated Social Media News in Banglacited by 3

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understandingcited by 9
  2. 2024'Since Lawyers are Males..': Examining Implicit Gender Bias in Hindi Language Generation by LLMscited by 7
  3. 2025IndoNLP 2025: Shared Task on Real-Time Reverse Transliteration for Romanized Indo-Aryan languagescited by 3
  4. 2025Beyond Specialization: Benchmarking LLMs for Transliteration of Indian Languagescited by 0