# INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages

arXiv: https://arxiv.org/abs/2407.13522 · HTML: https://mlsanity.com/paper/2407.13522/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://mlsanity.com/paper/2405.15032/) (2024)
- [XOR QA: Cross-lingual Open-Retrieval Question Answering](https://mlsanity.com/paper/2010.11856/) (2020)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [Do Multilingual Language Models Think Better in English?](https://mlsanity.com/paper/2308.01223/) (2023)
- [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024)
- [IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages](https://mlsanity.com/paper/2403.06350/) (2024)
- [Airavata: Introducing Hindi Instruction-tuned LLM](https://mlsanity.com/paper/2401.15006/) (2024)
- [English Machine Reading Comprehension Datasets: A Survey](https://mlsanity.com/paper/2101.10421/) (2021)
- [Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?](https://mlsanity.com/paper/2403.04792/) (2024)
- [On the Role of Parallel Data in Cross-lingual Transfer Learning](https://mlsanity.com/paper/2212.10173/) (2022)

## Built on by (most cited first, 24 of 24)

- [Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus](https://mlsanity.com/paper/2410.14815/) (2024)
- [Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages](https://mlsanity.com/paper/2411.12240/) (2024)
- [L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Context](https://mlsanity.com/paper/2409.08706/) (2024)
- [PARAM-1 BharatGen 2.9B Model](https://mlsanity.com/paper/2507.13390/) (2025)
- [Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges](https://mlsanity.com/paper/2509.11570/) (2025)
- [IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languages](https://mlsanity.com/paper/2505.03688/) (2025)
- [Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real World Setting](https://mlsanity.com/paper/2512.10780/) (2025)
- [Multilingual State Space Models for Structured Question Answering in Indic Languages](https://mlsanity.com/paper/2502.01673/) (2025)
- [ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjects](https://mlsanity.com/paper/2508.16185/) (2025)
- [IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages](https://mlsanity.com/paper/2512.00333/) (2025)
- [Long-context Non-factoid Question Answering in Indic Languages](https://mlsanity.com/paper/2504.13615/) (2025)
- [Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis](https://mlsanity.com/paper/2508.19831/) (2025)
- [ELR-1000: A Community-Generated Dataset for Endangered Indic Indigenous Languages](https://mlsanity.com/paper/2512.01077/) (2025)
- [Tourism Question Answer System in Indian Language using Domain-Adapted Foundation Models](https://mlsanity.com/paper/2511.23235/) (2025)
- [Language Models Entangle Language and Culture](https://mlsanity.com/paper/2601.15337/) (2026)
- [OmniEvalKit: A Modular, Lightweight Toolbox for Evaluating Large Language Model and its Omni-Extensions](https://mlsanity.com/paper/2412.06693/) (2024)
- [Eka-Eval: An Evaluation Framework for Low-Resource Multilingual Large Language Models](https://mlsanity.com/paper/2507.01853/) (2025)
- [Adapting Small Language Models to Low-Resource Domains: A Case Study in Hindi Tourism QA](https://mlsanity.com/paper/2510.25273/) (2025)
- [LittiChoQA: Literary Texts in Indic Languages Chosen for Question Answering](https://mlsanity.com/paper/2601.03025/) (2026)
- [A P=aninian Foundation for Indic Language Processing](https://mlsanity.com/paper/2606.24172/) (2026)
- [L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages](https://mlsanity.com/paper/2608.15535/) (2026)
- [Gurukul AI: An Interactive AI-Driven Educational Platform for Indian Education System](https://mlsanity.com/paper/2608.28611/) (2026)
- [MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain](https://mlsanity.com/paper/2609.04842/) (2026)
- [V=akQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering](https://mlsanity.com/paper/2609.19879/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) perform well on unseen tasks in English, but their abilities in non English languages are less explored due to limited benchmarks and training data. To bridge this gap, we introduce the Indic QA Benchmark, a large dataset for context grounded question answering in 11 major Indian languages, covering both extractive and abstractive tasks. Evaluations of multilingual LLMs, including instruction finetuned versions, revealed weak performance in low resource languages due to a strong English language bias in their training data. We also investigated the Translate Test paradigm,where inputs are translated to English for processing and the results are translated back into the source language for output. This approach outperformed multilingual LLMs, particularly in low resource settings. By releasing Indic QA, we aim to promote further research into LLMs question answering capabilities in low resource languages. This benchmark offers a critical resource to address existing limitations and foster multilingual understanding.