mlsanityData to 14 Oct 2026

VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan

arXiv 2503.10427 ↗First posted 13 Mar 2025cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

In this paper, we propose a comprehensive evaluation benchmark for Visual Language Models (VLM) in Traditional Chinese. Our evaluation suite, the first of its kind, contains two complementary components: (1) VisTW-MCQ, a collection of manually curated exam multi-choice questions from 21 academic subjects designed to test the broad knowledge and reasoning capabilities of VLMs; and (2) VisTW-Dialogue, an open dialogue benchmark comprising 131 image-question pairs manually created to evaluate VLMs' ability in free-form dialogue generation within Taiwanese cultural contexts. These benchmarks address a critical gap in the evaluation landscape, where existing benchmarks predominantly focus on English or Simplified Chinese, neglecting the unique linguistic and cultural aspects of Traditional Chinese used in regions like Taiwan and Hong Kong. Our analysis reveals significant performance differences across various VLMs and highlights specific challenges in processing Traditional Chinese visual content.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 32 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

23 of 32 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages25 Nov
  2. 2024JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation22 Oct
  3. 2024WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines16 Oct
  4. 2024CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation1 Jul
  5. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models15 Mar
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 17

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  3. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,887
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  7. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  8. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  10. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 195
  11. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
  12. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
5 more
  1. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
  2. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoningcited by 31
  3. 2024EVLM: An Efficient Vision-Language Model for Visual Understandingcited by 25
  4. 2023Advancing the Evaluation of Traditional Chinese Language Models: Towards a Comprehensive Benchmark Suitecited by 14
  5. 2025The Breeze 2 Herd of Models: Traditional Chinese LLMs Based on Llama with Vision-Aware and Function-Calling Capabilitiescited by 4

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Rice-VL: Evaluating Vision-Language Models for Cultural Understanding Across ASEAN Countriescited by 2
  2. 2025TAU: A Benchmark for Cultural Sound Understanding Beyond Semanticscited by 1
  3. 2025Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwancited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025"See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Models
  2. 2025TCC-Bench: Benchmarking the Traditional Chinese Culture Understanding Capabilities of MLLMs
  3. 2024AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models
  4. 2024Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanese
  5. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
  6. 2026ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China
  7. 2024GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation
  8. 2026VULCA-Bench: A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding