# MILU: A Multi-task Indic Language Understanding Benchmark

arXiv: https://arxiv.org/abs/2411.02538 · HTML: https://mlsanity.com/paper/2411.02538/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2024.

## Built on by (most cited first, 27 of 27)

- [Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia](https://mlsanity.com/paper/2503.07920/) (2025)
- [HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMs](https://mlsanity.com/paper/2503.12440/) (2025)
- [SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala](https://mlsanity.com/paper/2509.03162/) (2025)
- [Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges](https://mlsanity.com/paper/2509.11570/) (2025)
- [Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settings](https://mlsanity.com/paper/2509.24506/) (2025)
- [Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Study](https://mlsanity.com/paper/2507.14304/) (2025)
- [MUTANT: A Recipe for Multilingual Tokenizer Design](https://mlsanity.com/paper/2511.03237/) (2025)
- [ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjects](https://mlsanity.com/paper/2508.16185/) (2025)
- [IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages](https://mlsanity.com/paper/2512.00333/) (2025)
- [Common to Whom? Regional Cultural Commonsense and LLM Bias in India](https://mlsanity.com/paper/2601.15550/) (2026)
- [Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis](https://mlsanity.com/paper/2508.19831/) (2025)
- [ELR-1000: A Community-Generated Dataset for Endangered Indic Indigenous Languages](https://mlsanity.com/paper/2512.01077/) (2025)
- [Anthropogenic Regional Adaptation in Multimodal Vision-Language Model](https://mlsanity.com/paper/2604.11490/) (2026)
- [Better To Ask in English? Evaluating Factual Accuracy of Multilingual LLMs in English and Low-Resource Languages](https://mlsanity.com/paper/2504.20022/) (2025)
- [Test Set Quality in Multilingual LLM Evaluation](https://mlsanity.com/paper/2508.02635/) (2025)
- [Eka-Eval: An Evaluation Framework for Low-Resource Multilingual Large Language Models](https://mlsanity.com/paper/2507.01853/) (2025)
- [Intent Aware Context Retrieval for Multi-Turn Agricultural Question Answering](https://mlsanity.com/paper/2508.03719/) (2025)
- [MATA: Mindful Assessment of the Telugu Abilities of Large Language Models](https://mlsanity.com/paper/2508.13526/) (2025)
- [HiMed: Incentivizing Hindi Reasoning in Medical LLMs](https://mlsanity.com/paper/2605.24635/) (2026)
- [DEPART: DEcomposing PARiTy across Multilingual LLMs](https://mlsanity.com/paper/2605.28163/) (2026)
- [UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding](https://mlsanity.com/paper/2606.07167/) (2026)
- [The Masked Advantage: Uncovering Local-Language Access to Cultural Knowledge in LLMs](https://mlsanity.com/paper/2606.07422/) (2026)
- [Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension](https://mlsanity.com/paper/2608.03494/) (2026)
- [L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages](https://mlsanity.com/paper/2608.15535/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)
- [IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking](https://mlsanity.com/paper/2609.29167/) (2026)
- [In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B](https://mlsanity.com/paper/2610.15373/) (2026)

## Abstract (arXiv)

Evaluating Large Language Models (LLMs) in low-resource and linguistically diverse languages remains a significant challenge in NLP, particularly for languages using non-Latin scripts like those spoken in India. Existing benchmarks predominantly focus on English, leaving substantial gaps in assessing LLM capabilities in these languages. We introduce MILU, a Multi task Indic Language Understanding Benchmark, a comprehensive evaluation benchmark designed to address this gap. MILU spans 8 domains and 41 subjects across 11 Indic languages, reflecting both general and culturally specific knowledge. With an India-centric design, incorporates material from regional and state-level examinations, covering topics such as local history, arts, festivals, and laws, alongside standard subjects like science and mathematics. We evaluate over 42 LLMs, and find that current LLMs struggle with MILU, with GPT-4o achieving the highest average accuracy at 74 percent. Open multilingual models outperform language-specific fine-tuned models, which perform only slightly better than random baselines. Models also perform better in high resource languages as compared to low resource ones. Domain-wise analysis indicates that models perform poorly in culturally relevant areas like Arts and Humanities, Law and Governance compared to general fields like STEM. To the best of our knowledge, MILU is the first of its kind benchmark focused on Indic languages, serving as a crucial step towards comprehensive cultural evaluation. All code, benchmarks, and artifacts are publicly available to foster open research.