mlsanityData to 7 Oct 2026

IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages

arXiv 2404.16816 ↗First posted 25 Apr 2024cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) see increasing adoption across the globe, it is imperative for LLMs to be representative of the linguistic diversity of the world. India is a linguistically diverse country of 1.4 Billion people. To facilitate research on multilingual LLM evaluation, we release IndicGenBench - the largest benchmark for evaluating LLMs on user-facing generation tasks across a diverse set 29 of Indic languages covering 13 scripts and 4 language families. IndicGenBench is composed of diverse generation tasks like cross-lingual summarization, machine translation, and cross-lingual question answering. IndicGenBench extends existing benchmarks to many Indic languages through human curation providing multi-way parallel evaluation data for many under-represented Indic languages for the first time. We evaluate a wide range of proprietary and open-source LLMs including GPT-3.5, GPT-4, PaLM-2, mT5, Gemma, BLOOM and LLaMA on IndicGenBench in a variety of settings. The largest PaLM-2 models performs the best on most tasks, however, there is a significant performance gap in all languages compared to English showing that further research is needed for the development of more inclusive multilingual language models. IndicGenBench is released at www.github.com/google-research-datasets/indic-gen-bench

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 37 of its 46 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

37 of 46 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  12. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
25 more
  1. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  2. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  3. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  4. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  5. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  6. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  7. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  8. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  9. 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
  10. 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 149
  11. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 133
  12. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  13. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 101
  14. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  15. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  16. 2020A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matterscited by 45
  17. 2022IndicNLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in Indic Languagescited by 42
  18. 2024MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modelingcited by 40
  19. 2021CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairscited by 39
  20. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  21. 2024Airavata: Introducing Hindi Instruction-tuned LLMcited by 32
  22. 2023Tamil-Llama: A New Tamil Language Model Based on Llama 2cited by 31
  23. 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
  24. 2023PMIndiaSum: Multilingual and Cross-lingual Headline Summarization for Languages in Indiacited by 11
  25. 2023Mukhyansh: A Headline Generation Dataset for Indic Languagescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 29; 2026: 15.

55 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  3. 2024INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languagescited by 24
  4. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 17
  5. 2025SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culturecited by 14
  6. 2024Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languagescited by 10
  7. 2025Chitrarth: Bridging Vision and Language for a Billion Peoplecited by 10
  8. 2025Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindicited by 10
  9. 2025TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Storiescited by 10
  10. 2024Efficient Continual Pre-training of LLMs for Low-resource Languagescited by 8
  11. 2024Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languagescited by 7
  12. 2025Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examplescited by 6
38 more
  1. 2024Generalists vs. Specialists: Evaluating Large Language Models for Urducited by 5
  2. 2024PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Bankscited by 5
  3. 2025Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluatorscited by 5
  4. 2025TLUE: A Tibetan Language Understanding Evaluation Benchmarkcited by 5
  5. 2025IberBench: LLM Evaluation on Iberian Languagescited by 5
  6. 2025Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real World Settingcited by 5
  7. 2026IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languagescited by 5
  8. 2024UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Modelscited by 4
  9. 2024Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancementcited by 4
  10. 2025MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languagescited by 4
  11. 2025UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languagescited by 4
  12. 2025DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linkingcited by 3
  13. 2025Pragyaan: Designing and Curating High-Quality Cultural Post-Training Datasets for Indian Languagescited by 3
  14. 2025MULTITAT: Benchmarking Multilingual Table-and-Text Question Answeringcited by 2
  15. 2025COSMMIC: Comment-Sensitive Multimodal Multilingual Indian Corpus for Summarization and Headline Generationcited by 2
  16. 2025ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjectscited by 2
  17. 2025Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysiscited by 2
  18. 2025IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languagescited by 2
  19. 2024SSP: Self-Supervised Prompting for Cross-Lingual Transfer to Low-Resource Languages using Large Language Modelscited by 1
  20. 2025Better To Ask in English? Evaluating Factual Accuracy of Multilingual LLMs in English and Low-Resource Languagescited by 1
  21. 2025Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarkscited by 1
  22. 2025Leveraging the Cross-Domain & Cross-Linguistic Corpus for Low Resource NMT: A Case Study On Bhili-Hindi-English Parallel Corpuscited by 1
  23. 2025The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs using Indian Riddlescited by 1
  24. 2025ELR-1000: A Community-Generated Dataset for Endangered Indic Indigenous Languagescited by 1
  25. 2026GaelEval: Benchmarking LLM Performance for Scottish Gaeliccited by 1
  26. 2026Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluationcited by 1
  27. 2026IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languagescited by 1
  28. 2025Eka-Eval: An Evaluation Framework for Low-Resource Multilingual Large Language Modelscited by 0
  29. 2025Uncertainty-driven Embedding Convolutioncited by 0
  30. 2025MATA: Mindful Assessment of the Telugu Abilities of Large Language Modelscited by 0
  31. 2025Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extractioncited by 0
  32. 2025HinTel-AlignBench: A Framework and Benchmark for Hindi-Telugu with English-Aligned Samplescited by 0
  33. 2026The Language of Bargaining: Linguistic Effects in LLM Negotiationscited by 0
  34. 2026MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generationcited by 0
  35. 2026The Annotation Scarcity Paradox in Low-Resource NLP Evaluation: A Decade of Acceleration and Emerging Constraintscited by 0
  36. 2026BhashaSetu: A Data-Centric Approach to Low-Resource Machine Translationcited by 0
  37. 2026Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferralcited by 0
  38. 2026Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extensioncited by 0

50 most cited of 55 All 55 (JSON)