# VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan

arXiv: https://arxiv.org/abs/2503.10427 · HTML: https://mlsanity.com/paper/2503.10427/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [ColPali: Efficient Document Retrieval with Vision Language Models](https://mlsanity.com/paper/2407.01449/) (2024)
- [The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink](https://mlsanity.com/paper/2204.05149/) (2022)
- [M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models](https://mlsanity.com/paper/2306.05179/) (2023)
- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models](https://mlsanity.com/paper/2403.10378/) (2024)
- [WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines](https://mlsanity.com/paper/2410.12705/) (2024)
- [Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models](https://mlsanity.com/paper/2405.02287/) (2024)
- [CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning](https://mlsanity.com/paper/2401.14011/) (2024)
- [EVLM: An Efficient Vision-Language Model for Visual Understanding](https://mlsanity.com/paper/2407.14177/) (2024)
- [JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation](https://mlsanity.com/paper/2410.17250/) (2024)
- [Advancing the Evaluation of Traditional Chinese Language Models: Towards a Comprehensive Benchmark Suite](https://mlsanity.com/paper/2309.08448/) (2023)
- [CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation](https://mlsanity.com/paper/2407.01081/) (2024)
- [The Breeze 2 Herd of Models: Traditional Chinese LLMs Based on Llama with Vision-Aware and Function-Calling Capabilities](https://mlsanity.com/paper/2501.13921/) (2025)

## Built on by (most cited first, 3 of 3)

- [Rice-VL: Evaluating Vision-Language Models for Cultural Understanding Across ASEAN Countries](https://mlsanity.com/paper/2512.01419/) (2025)
- [TAU: A Benchmark for Cultural Sound Understanding Beyond Semantics](https://mlsanity.com/paper/2509.26329/) (2025)
- [Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan](https://mlsanity.com/paper/2508.01274/) (2025)

## Abstract (arXiv)

In this paper, we propose a comprehensive evaluation benchmark for Visual Language Models (VLM) in Traditional Chinese. Our evaluation suite, the first of its kind, contains two complementary components: (1) VisTW-MCQ, a collection of manually curated exam multi-choice questions from 21 academic subjects designed to test the broad knowledge and reasoning capabilities of VLMs; and (2) VisTW-Dialogue, an open dialogue benchmark comprising 131 image-question pairs manually created to evaluate VLMs' ability in free-form dialogue generation within Taiwanese cultural contexts. These benchmarks address a critical gap in the evaluation landscape, where existing benchmarks predominantly focus on English or Simplified Chinese, neglecting the unique linguistic and cultural aspects of Traditional Chinese used in regions like Taiwan and Hong Kong. Our analysis reveals significant performance differences across various VLMs and highlights specific challenges in processing Traditional Chinese visual content.