# HinTel-AlignBench: A Framework and Benchmark for Hindi-Telugu with English-Aligned Samples

arXiv: https://arxiv.org/abs/2511.15183 · HTML: https://mlsanity.com/paper/2511.15183/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Builds on (most cited first, 13 of 13)

- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://mlsanity.com/paper/2305.16307/) (2023)
- [CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning](https://mlsanity.com/paper/2208.05358/) (2022)
- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [xGQA: Cross-Lingual Visual Question Answering](https://mlsanity.com/paper/2109.06082/) (2021)
- [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024)
- [PALO: A Polyglot Large Multimodal Model for 5B People](https://mlsanity.com/paper/2402.14818/) (2024)
- [The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks](https://mlsanity.com/paper/2504.15521/) (2025)
- [IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding](https://mlsanity.com/paper/2501.15747/) (2025)
- [Chitrarth: Bridging Vision and Language for a Billion People](https://mlsanity.com/paper/2502.15392/) (2025)

## Abstract (arXiv)

With nearly 1.5 billion people and more than 120 major languages, India represents one of the most diverse regions in the world. As multilingual Vision-Language Models (VLMs) gain prominence, robust evaluation methodologies are essential to drive progress toward equitable AI for low-resource languages. Current multilingual VLM evaluations suffer from four major limitations: reliance on unverified auto-translations, narrow task/domain coverage, limited sample sizes, and lack of cultural and natively sourced Question-Answering (QA). To address these gaps, we present a scalable framework to evaluate VLMs in Indian languages and compare it with performance in English. Using the framework, we generate HinTel-AlignBench, a benchmark that draws from diverse sources in Hindi and Telugu with English-aligned samples. Our contributions are threefold: (1) a semi-automated dataset creation framework combining back-translation, filtering, and human verification; (2) the most comprehensive vision-language benchmark for Hindi and and Telugu, including adapted English datasets (VQAv2, RealWorldQA, CLEVR-Math) and native novel Indic datasets (JEE for STEM, VAANI for cultural grounding) with approximately 4,000 QA pairs per language; and (3) a detailed performance analysis of various State-of-the-Art (SOTA) open-weight and closed-source VLMs. We find a regression in performance for tasks in English versus in Indian languages for 4 out of 5 tasks across all the models, with an average regression of 8.3 points in Hindi and 5.5 points for Telugu. We categorize common failure modes to highlight concrete areas of improvement in multilingual multimodal understanding.