mlsanityData to 7 Oct 2026

L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Context

arXiv 2409.08706 ↗First posted 13 Sep 2024cs.CL cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have made significant progress in incorporating Indic languages within multilingual models. However, it is crucial to quantitatively assess whether these languages perform comparably to globally dominant ones, such as English. Currently, there is a lack of benchmark datasets specifically designed to evaluate the regional knowledge of LLMs in various Indic languages. In this paper, we present the L3Cube-IndicQuest, a gold-standard factual question-answering benchmark dataset designed to evaluate how well multilingual LLMs capture regional knowledge across various Indic languages. The dataset contains 200 question-answer pairs, each for English and 19 Indic languages, covering five domains specific to the Indic region. We aim for this dataset to serve as a benchmark, providing ground truth for evaluating the performance of LLMs in understanding and representing knowledge relevant to the Indian context. The IndicQuest can be used for both reference-based evaluation and LLM-as-a-judge evaluation. The dataset is shared publicly at https://github.com/l3cube-pune/indic-nlp .

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 9 of its 15 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

9 of 15 references.

References · 9

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  4. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  5. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  6. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  7. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  8. 2022L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Librarycited by 30
  9. 2024INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languagescited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 3.

10 papers.

  1. 2024Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpuscited by 24
  2. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
  3. 2025IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languagescited by 4
  4. 2025Pragyaan: Designing and Curating High-Quality Cultural Post-Training Datasets for Indian Languagescited by 3
  5. 2026Common to Whom? Regional Cultural Commonsense and LLM Bias in Indiacited by 3
  6. 2025Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysiscited by 2
  7. 2025Better To Ask in English? Evaluating Factual Accuracy of Multilingual LLMs in English and Low-Resource Languagescited by 1
  8. 2025Leveraging Synthetic Data for Question Answering with Multilingual LLMs in the Agricultural Domaincited by 1
  9. 2026L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languagescited by 0
  10. 2026V=akQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answeringcited by 0