mlsanityData to 14 Oct 2026

Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan

arXiv 2508.01274 ↗First posted 2 Aug 2025cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) process visual, acoustic, and textual inputs, addressing the limitations of single-modality LLMs. However, existing benchmarks often overlook tri-modal evaluation in Traditional Chinese and do not consider inference latency. To address this, we introduce Multi-TW, the first Traditional Chinese benchmark for evaluating the performance and latency of any-to-any multimodal models. Multi-TW includes 900 multiple-choice questions (image and text, audio and text pairs) sourced from official proficiency tests developed with the Steering Committee for the Test of Proficiency-Huayu (SC-TOP). We evaluated various any-to-any models and vision-language models (VLMs) with audio transcription. Our results show that closed-source models generally outperform open-source ones across modalities, although open-source models can perform well in audio tasks. End-to-end any-to-any pipelines offer clear latency advantages compared to VLMs using separate audio transcription. Multi-TW presents a comprehensive view of model capabilities and highlights the need for Traditional Chinese fine-tuning and efficient multimodal architectures.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 27 references that are in the map. First posted on arXiv 2 Aug 2025.

Builds on

14 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  2. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep

Other references · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  7. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  8. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 207
  9. 2023Sparks of Large Audio Models: A Survey and Outlookcited by 41
  10. 2023Advancing the Evaluation of Traditional Chinese Language Models: Towards a Comprehensive Benchmark Suitecited by 14
  11. 2024An Improved Traditional Chinese Evaluation Suite for Foundation Modelcited by 13
  12. 2025VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwancited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation
  2. 2025ChineseVideoBench: Benchmarking Multi-modal Large Models for Chinese Video Question Answering
  3. 2025LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation
  4. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  5. 2024AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models
  6. 2025TCC-Bench: Benchmarking the Traditional Chinese Culture Understanding Capabilities of MLLMs
  7. 2024A Survey on Benchmarks of Multimodal Large Language Models
  8. 2024MULTI: Multimodal Understanding Leaderboard with Text and Images