# IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages

arXiv: https://arxiv.org/abs/2606.19157 · HTML: https://mlsanity.com/paper/2606.19157/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 20 of 20)

- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Cold Fusion: Training Seq2Seq Models Together with Language Models](https://mlsanity.com/paper/1708.06426/) (2017)
- [Earnings-22: A Practical Benchmark for Accents in the Wild](https://mlsanity.com/paper/2203.15591/) (2022)
- [Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition](https://mlsanity.com/paper/2306.16007/) (2023)
- [Voxtral](https://mlsanity.com/paper/2507.13264/) (2025)
- [IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages](https://mlsanity.com/paper/2403.01926/) (2024)
- [Can Contextual Biasing Remain Effective with Whisper and GPT-2?](https://mlsanity.com/paper/2306.01942/) (2023)
- [Improving Contextual Recognition of Rare Words with an Alternate Spelling Prediction Model](https://mlsanity.com/paper/2209.01250/) (2022)
- [Improving ASR Contextual Biasing with Guided Attention](https://mlsanity.com/paper/2401.08835/) (2024)
- [ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark](https://mlsanity.com/paper/2507.05727/) (2025)
- [Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whisper](https://mlsanity.com/paper/2406.05806/) (2024)
- [BR-ASR: Efficient and Scalable Bias Retrieval Framework for Contextual Biasing ASR in Speech LLM](https://mlsanity.com/paper/2505.19179/) (2025)
- [Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model](https://mlsanity.com/paper/2410.18363/) (2024)
- [SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition](https://mlsanity.com/paper/2409.10429/) (2024)
- [PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech](https://mlsanity.com/paper/2512.23686/) (2025)
- [Improving Rare-Word Recognition of Whisper in Zero-Shot Settings](https://mlsanity.com/paper/2502.11572/) (2025)
- [Lightweight Prompt Biasing for Contextualized End-to-End ASR Systems](https://mlsanity.com/paper/2506.06252/) (2025)

## Built on by (most cited first, 3 of 3)

- [No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus](https://mlsanity.com/paper/2608.28875/) (2026)
- [HearInContext: A Benchmark for Implicit Context in Speech Recognition](https://mlsanity.com/paper/2609.18680/) (2026)
- [IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking](https://mlsanity.com/paper/2609.29167/) (2026)

## Abstract (arXiv)

AudioLLMs enable speech recognition conditioned on textual prompts such as domain descriptions or entity lists. However, it remains unclear whether these models genuinely utilise such context or rely on parametric knowledge learned during pretraining. Existing benchmarks cannot answer this question because they evaluate transcription under fixed prompting conditions and rarely include explicit contextual inputs. We introduce IndicContextEval, a 56-hour multilingual benchmark of natural speech from 555 speakers across 8 Indian languages and 23 professional domains. We design a 7-level prompting framework that progressively introduces contextual signals, including metadata, natural-language descriptions, entity lists in English and native script, and adversarial prompts with incorrect entities. Evaluating five models reveals substantial differences in context utilisation behaviour, highlighting the need for explicit evaluation of contextual grounding in AudioLLMs.