mlsanityData to 14 Oct 2026

DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture

arXiv 2509.19274 ↗First posted 23 Sep 2025cs.CL cs.MMcited by 13

Not on the map. Why?

Abstract (arXiv)

We introduce DRISHTIKON, a first-of-its-kind multimodal and multilingual benchmark centered exclusively on Indian culture, designed to evaluate the cultural understanding of generative AI systems. Unlike existing benchmarks with a generic or global scope, DRISHTIKON offers deep, fine-grained coverage across India's diverse regions, spanning 15 languages, covering all states and union territories, and incorporating over 64,000 aligned text-image pairs. The dataset captures rich cultural themes including festivals, attire, cuisines, art forms, and historical heritage amongst many more. We evaluate a wide range of vision-language models (VLMs), including open-source small and large models, proprietary systems, reasoning-specialized VLMs, and Indic-focused models, across zero-shot and chain-of-thought settings. Our results expose key limitations in current models' ability to reason over culturally grounded, multimodal inputs, particularly for low-resource languages and less-documented traditions. DRISHTIKON fills a vital gap in inclusive AI research, offering a robust testbed to advance culturally aware, multimodally competent language technologies.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

45 of 62 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  11. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  12. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
33 more
  1. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  2. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  4. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  5. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 173
  6. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  7. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 103
  8. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 102
  9. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 101
  10. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  11. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 62
  12. 2024How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensionscited by 52
  13. 2024Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsensecited by 51
  14. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 39
  15. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
  16. 2024IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Contextcited by 32
  17. 2024NativQA: Multilingual Culturally-Aligned Natural Query for LLMscited by 30
  18. 2024M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Taskscited by 27
  19. 2024LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Outputcited by 27
  20. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 26
  21. 2024DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subculturescited by 24
  22. 2020Indic-Transformers: An Analysis of Transformer Language Models for Indian Languagescited by 22
  23. 2024MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English Clinical Queriescited by 19
  24. 2024JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluationcited by 19
  25. 2025GIMMICK -- Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarkingcited by 16
  26. 2025SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culturecited by 16
  27. 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Modelscited by 14
  28. 2023Large Scale Multi-Lingual Multi-Modal Summarization Datasetcited by 12
  29. 2024L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Contextcited by 10
  30. 2025Chitrarth: Bridging Vision and Language for a Billion Peoplecited by 10
  31. 2024The World Wide recipe: A community-centred framework for fine-grained data collection and regional bias operationalisationcited by 7
  32. 2022Cultural Re-contextualization of Fairness Research in Language Technologies in Indiacited by 4
  33. 2025COSMMIC: Comment-Sensitive Multimodal Multilingual Indian Corpus for Summarization and Headline Generationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcarecited by 9
  2. 2026CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcarecited by 4
  3. 2025IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languagescited by 3
  4. 2026Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialectscited by 3
  5. 2025Camellia: Benchmarking Cultural Biases in LLMs for Asian Languagescited by 2
  6. 2026Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluationcited by 1
  7. 2026Toward Culturally Grounded Natural Language Processingcited by 0
  8. 2026Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languagescited by 0
  9. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contextscited by 0
  10. 2026When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Modelscited by 0
  11. 2026ArtiFact: A Large-Scale Multi-Modal Cultural Heritage Datasetcited by 0
  12. 2026MET: Theory-Grounded and Culture-Aware Multilingual Moral Reasoningcited by 0
1 more
  1. 2026CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
  2. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
  3. 2025Seeing Culture: A Benchmark for Visual Reasoning and Grounding
  4. 2026Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages
  5. 2025EgMM-Corpus: A Multimodal Vision-Language Dataset for Egyptian Culture
  6. 2026MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models
  7. 2025Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation
  8. 2025When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contexts