mlsanityData to 13 Oct 2026

VakyArth: Evaluating Pragmatic Competence in LLMs across Indic Languages

arXiv 2609.01788 ↗First posted 1 Sep 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Real-world communication often requires pragmatic reasoning: interpreting meanings implied through context and cultural convention rather than stated literally. Existing pragmatic evaluation remains largely limited to English and high-resource languages, leaving Indic languages unexplored despite their linguistic and cultural diversity. We introduce VakyArth, the first pragmatic benchmark for Indic languages, designed as a diagnostic evaluation covering Hindi, Punjabi, Tamil, and Malayalam. VakyArth evaluates models across five phenomena: deixis, speech acts, implicature, social pragmatics, and coherence; through multiple-choice questions, natural language inference, and translation, with all items authored by native speakers. Across multilingual large language models (LLMs) of varying families and sizes, we find consistent failures on pragmatic meanings rooted in Indic linguistic and cultural conventions. Our analysis shows systematic differences across languages and tasks: MCQ accuracy exceeds NLI accuracy in all model-language combinations, translation performance does not reliably track pragmatic understanding, and Indo-Aryan languages show a translation advantage over Dravidian languages. We further show that automatic translation metrics can miss fluent but pragmatically unfaithful outputs, especially for implicature and deixis.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

17 of 28 references.

References · 17

  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,027
  2. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 414
  3. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  4. 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 76
  5. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 58
  6. 2022The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMscited by 57
  7. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 55
  8. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  9. 2020Processing South Asian Languages Written in the Latin Script: the Dakshina Datasetcited by 35
  10. 2025Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challengescited by 31
  11. 2024RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanizationcited by 29
  12. 2025On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomenacited by 21
5 more
  1. 2023Modeling Cross-Cultural Pragmatic Inference with Codenames Duetcited by 20
  2. 2021Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Modelscited by 15
  3. 2024Pragmatic Competence Evaluation of Large Language Models for the Korean Languagecited by 8
  4. 2025Evaluating Large language models on Understanding Korean indirect Speech actscited by 3
  5. 2025The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Modelscited by 3

Built on by

Not cited yet.