# GIMMICK -- Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking

arXiv: https://arxiv.org/abs/2502.13766 · HTML: https://mlsanity.com/paper/2502.13766/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 19 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Built on by (most cited first, 16 of 16)

- [DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture](https://mlsanity.com/paper/2509.19274/) (2025)
- [Seeing Culture: A Benchmark for Visual Reasoning and Grounding](https://mlsanity.com/paper/2509.16517/) (2025)
- [Pearl: A Multimodal Culturally-Aware Arabic Instruction Dataset](https://mlsanity.com/paper/2505.21979/) (2025)
- [Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responses](https://mlsanity.com/paper/2601.13024/) (2026)
- [Uncovering Cultural Representation Disparities in Vision-Language Models](https://mlsanity.com/paper/2505.14729/) (2025)
- [Identity-Aware Large Language Models require Cultural Reasoning](https://mlsanity.com/paper/2510.18510/) (2025)
- [Evaluation of Cultural Competence of Vision-Language Models](https://mlsanity.com/paper/2505.22793/) (2025)
- [VULCA-Bench: A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding](https://mlsanity.com/paper/2601.07986/) (2026)
- [Culture In a Frame: C3B as a Comic-Based Benchmark for Multimodal Culturally Awareness](https://mlsanity.com/paper/2510.00041/) (2025)
- [On The Conceptualization and Societal Impact of Cross-Cultural Bias](https://mlsanity.com/paper/2512.21809/) (2025)
- [Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models](https://mlsanity.com/paper/2601.07984/) (2026)
- [ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs](https://mlsanity.com/paper/2604.06484/) (2026)
- [Toward Culturally Grounded Natural Language Processing](https://mlsanity.com/paper/2603.26013/) (2026)
- [Cultural Moment Benchmark: Evaluating Video Cultural Reasoning and Grounding in Southeast Asia](https://mlsanity.com/paper/2608.23065/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)
- [ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models](https://mlsanity.com/paper/2610.20472/) (2026)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) have recently gained attention due to their distinctive performance and broad applicability. While it has been previously shown that their efficacy in usage scenarios involving non-Western contexts falls short, existing studies are limited in scope, covering just a narrow range of cultures, focusing exclusively on a small number of cultural aspects, or evaluating a limited selection of models on a single task only. Towards globally inclusive LVLM research, we introduce GIMMICK, an extensive multimodal benchmark designed to assess a broad spectrum of cultural knowledge across 144 countries representing six global macro-regions. GIMMICK comprises six tasks built upon three new datasets that span 728 unique cultural events or facets on which we evaluated 20 LVLMs and 11 LLMs, including five proprietary and 26 open-weight models of all sizes. We systematically examine (1) regional cultural biases, (2) the influence of model size, (3) input modalities, and (4) external cues. Our analyses reveal strong biases toward Western cultures across models and tasks and highlight strong correlations between model size and performance, as well as the effectiveness of multimodal input and external geographic cues. We further find that models have more knowledge of tangible than intangible aspects (e.g., food vs. rituals) and that they excel in recognizing broad cultural origins but struggle with a more nuanced understanding.