IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages
Not on the map. Why?
Abstract (arXiv)
Instruction-following benchmarks remain predominantly English-centric, leaving a critical evaluation gap for the hundreds of millions of Indic language speakers. We introduce IndicIFEval, a benchmark evaluating constrained generation of LLMs across 14 Indic languages using automatically verifiable, rule-based instructions. It comprises around 800 human-verified examples per language spread across two complementary subsets: IndicIFEval-Ground, translated prompts from IFEval (Zhou et al., 2023) carefully localized for Indic contexts, and IndicIFEval-Ground, synthetically generated instructions grounded in native Indic content. We conduct a comprehensive evaluation of major open-weight and proprietary models spanning both reasoning and non-reasoning models. While models maintain strong adherence to formatting constraints, they struggle significantly with lexical and cross-lingual tasks -- and despite progress in high-resource languages, instruction-following across the broader Indic family lags significantly behind English. We release IndicIFEval and its evaluation scripts to support progress on multilingual constrained generation (http://github.com/ai4bharat/IndicIFEval).
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 13 of its 22 references that are in the map. First posted on arXiv 25 Feb 2026.
Builds on
13 of 22 references.
References · 13
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,191
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,163
- 2023Large Language Models are not Fair Evaluatorscited by 1,054
- 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 833
- 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
- 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
- 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
- 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
- 2024IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languagescited by 52
- 2025M-IFEval: Multilingual Instruction-Following Evaluationcited by 17
- 2025MaXIFE: Multilingual and Cross-lingual Instruction Following Evaluationcited by 7
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
- 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models
- 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
- 2026IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following
- 2026Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs
- 2026DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions
- 2025Training with Pseudo-Code for Instruction Following