mlsanityData to 7 Oct 2026

Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration

arXiv 2406.16469 ↗First posted 24 Jun 2024cs.CL cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

To create culturally inclusive vision-language models (VLMs), developing a benchmark that tests their ability to address culturally relevant questions is essential. Existing approaches typically rely on human annotators, making the process labor-intensive and creating a cognitive burden in generating diverse questions. To address this, we propose a semi-automated framework for constructing cultural VLM benchmarks, specifically targeting multiple-choice QA. This framework combines human-VLM collaboration, where VLMs generate questions based on guidelines, a small set of annotated examples, and relevant knowledge, followed by a verification process by native speakers. We demonstrate the effectiveness of this framework through the creation of textttK-Viscuit, a dataset focused on Korean culture. Our experiments on this dataset reveal that open-source models lag behind proprietary ones in understanding Korean culture, highlighting key areas for improvement. We also present a series of further analyses, including human evaluation, augmenting VLMs with external knowledge, and the evaluation beyond multiple-choice QA. Our dataset is available at https://huggingface.co/datasets/ddehun/k-viscuit.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 41 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

22 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods5 Jan

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  10. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  11. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  12. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
9 more
  1. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  2. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  3. 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 65
  4. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  5. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  6. 2023GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognitioncited by 50
  7. 2023COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuancescited by 35
  8. 2021Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistantscited by 31
  9. 2023Dataset Bias Mitigation in Multiple-Choice Visual Question Answering and Beyondcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 8.

19 papers.

Sort:

By area:

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 100
  3. 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Modelscited by 14
  4. 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thoughtcited by 10
  5. 2025Seeing Culture: A Benchmark for Visual Reasoning and Groundingcited by 8
  6. 2025Pearl: A Multimodal Culturally-Aware Arabic Instruction Datasetcited by 6
  7. 2025KRETA: A Benchmark for Korean Reading and Reasoning in Text-Rich VQA Attuned to Diverse Visual Contextscited by 6
  8. 2025VARCO-VISION-2.0 Technical Reportcited by 5
  9. 2025TCC-Bench: Benchmarking the Traditional Chinese Culture Understanding Capabilities of MLLMscited by 4
  10. 2025When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contextscited by 3
  11. 2025Evaluation of Cultural Competence of Vision-Language Modelscited by 3
  12. 2026What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Modelscited by 2
7 more
  1. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 1
  2. 2026KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examinationcited by 1
  3. 2026EXAONE 4.5 Technical Reportcited by 0
  4. 2026Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challengescited by 0
  5. 2026PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understandingcited by 0
  6. 2026Cultural Moment Benchmark: Evaluating Video Cultural Reasoning and Grounding in Southeast Asiacited by 0
  7. 2026CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
  2. 2025BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models
  3. 2025Rice-VL: Evaluating Vision-Language Models for Cultural Understanding Across ASEAN Countries
  4. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
  5. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts
  6. 2026VULCA-Bench: A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding
  7. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledge
  8. 2025MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation