# ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjects

arXiv: https://arxiv.org/abs/2508.16185 · HTML: https://mlsanity.com/paper/2508.16185/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 22 Aug 2025.

## Builds on (most cited first, 16 of 16)

- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions](https://mlsanity.com/paper/2309.12342/) (2023)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [KMMLU: Measuring Massive Multitask Language Understanding in Korean](https://mlsanity.com/paper/2402.11548/) (2024)
- [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024)
- [PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data](https://mlsanity.com/paper/2406.15053/) (2024)
- [MILU: A Multi-task Indic Language Understanding Benchmark](https://mlsanity.com/paper/2411.02538/) (2024)
- [INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages](https://mlsanity.com/paper/2407.13522/) (2024)
- [IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning](https://mlsanity.com/paper/2407.05399/) (2024)
- [SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture](https://mlsanity.com/paper/2506.15355/) (2025)
- [A Benchmark and Dataset for Post-OCR text correction in Sanskrit](https://mlsanity.com/paper/2211.07980/) (2022)
- [IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding](https://mlsanity.com/paper/2501.15747/) (2025)
- [PARAM-1 BharatGen 2.9B Model](https://mlsanity.com/paper/2507.13390/) (2025)
- [Gender Representation and Bias in Indian Civil Service Mock Interviews](https://mlsanity.com/paper/2409.12194/) (2024)

## Built on by (most cited first, 3 of 3)

- [IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages](https://mlsanity.com/paper/2512.00333/) (2025)
- [VIRAASAT: Traversing Novel Paths for Indian Cultural Reasoning](https://mlsanity.com/paper/2602.18429/) (2026)
- [BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources](https://mlsanity.com/paper/2604.18423/) (2026)

## Abstract (arXiv)

Large language models have been widely evaluated on tasks such as comprehension, summarization, code generation, etc. However, their performance on graduate-level, culturally grounded questions in the Indian context remains largely unexplored. Existing Indian benchmarks emphasise basic fact-orientated queries that offer limited assessment of a deeper disciplinary understanding tailored to the Indian setting. In this paper, we present ParamBench, consisting of more than 17K questions in the Hindi language, comprising questionnaires from 21 diverse subjects. These questions are primarily derived from a nationwide graduate-level entrance examination covering topics such as history, music, instruments, yoga, literature, philosophy, law, etc.~ specifically for the Indian context. Additionally, we assess the ability of LLMs to handle diverse question formats - such as list-based matching, assertion-reason pairs, and sequence ordering - alongside conventional multiple-choice questions. We evaluated the performance of more than 16 open source LLMs on this benchmark, observing that Gemma3-27B attains the highest overall accuracy of 56.4%. Furthermore, subject-wise analysis indicates that even for the best-performing LLMs, performance remains weak on topics such as music, classical instruments, and law, underscoring persistent challenges in culturally grounded reasoning. The dataset and source code is present at https://github.com/ayushbits/ParamBench.