mlsanityData to 14 Oct 2026

Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis

arXiv 2508.19831 ↗First posted 27 Aug 2025cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Evaluating instruction-tuned Large Language Models (LLMs) in Hindi is challenging due to a lack of high-quality benchmarks, as direct translation of English datasets fails to capture crucial linguistic and cultural nuances. To address this, we introduce a suite of five Hindi LLM evaluation datasets: IFEval-Hi, MT-Bench-Hi, GSM8K-Hi, ChatRAG-Hi, and BFCL-Hi. These were created using a methodology that combines from-scratch human annotation with a translate-and-verify process. We leverage this suite to conduct an extensive benchmarking of open-source LLMs supporting Hindi, providing a detailed comparative analysis of their current capabilities. Our curation process also serves as a replicable methodology for developing benchmarks in other low-resource languages.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 27 Aug 2025.

Builds on

23 of 30 references.

References · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
11 more
  1. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  2. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  3. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  4. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 102
  5. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 56
  6. 2024MILU: A Multi-task Indic Language Understanding Benchmarkcited by 27
  7. 2024INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languagescited by 24
  8. 2024Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpuscited by 24
  9. 2024L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Contextcited by 10
  10. 2025IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understandingcited by 10
  11. 2025IndicSQuAD: A Comprehensive Multilingual Question Answering Dataset for Indic Languagescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languagescited by 1
  2. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1