# BhashaBench V1: A Comprehensive Benchmark for the Quadrant of Indic Domains

arXiv: https://arxiv.org/abs/2510.25409 · HTML: https://mlsanity.com/paper/2510.25409/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2025.

## Built on by (most cited first, 7 of 7)

- [VIRAASAT: Traversing Novel Paths for Indian Cultural Reasoning](https://mlsanity.com/paper/2602.18429/) (2026)
- [FLARE: Task-agnostic embedding model evaluation through a normalization process](https://mlsanity.com/paper/2604.17344/) (2026)
- [HiMed: Incentivizing Hindi Reasoning in Medical LLMs](https://mlsanity.com/paper/2605.24635/) (2026)
- [Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering](https://mlsanity.com/paper/2607.19856/) (2026)
- [Can Legal AI Know When It Is Wrong? And Do Students Know When It Is?](https://mlsanity.com/paper/2608.21089/) (2026)
- [IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking](https://mlsanity.com/paper/2609.29167/) (2026)
- [AgriJev: Efficient Agricultural Candidate Scoring with Multi-Scale Adapted Models](https://mlsanity.com/paper/2610.19478/) (2026)

## Abstract (arXiv)

The rapid advancement of large language models(LLMs) has intensified the need for domain and culture specific evaluation. Existing benchmarks are largely Anglocentric and domain-agnostic, limiting their applicability to India-centric contexts. To address this gap, we introduce BhashaBench V1, the first domain-specific, multi-task, bilingual benchmark focusing on critical Indic knowledge systems. BhashaBench V1 contains 74,166 meticulously curated question-answer pairs, with 52,494 in English and 21,672 in Hindi, sourced from authentic government and domain-specific exams. It spans four major domains: Agriculture, Legal, Finance, and Ayurveda, comprising 90+ subdomains and covering 500+ topics, enabling fine-grained evaluation. Evaluation of 29+ LLMs reveals significant domain and language specific performance gaps, with especially large disparities in low-resource domains. For instance, GPT-4o achieves 76.49% overall accuracy in Legal but only 59.74% in Ayurveda. Models consistently perform better on English content compared to Hindi across all domains. Subdomain-level analysis shows that areas such as Cyber Law, International Finance perform relatively well, while Panchakarma, Seed Science, and Human Rights remain notably weak. BhashaBench V1 provides a comprehensive dataset for evaluating large language models across India's diverse knowledge domains. It enables assessment of models' ability to integrate domain-specific knowledge with bilingual understanding. All code, benchmarks, and resources are publicly available to support open research.