mlsanityData to 14 Oct 2026

HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Papers

arXiv 2606.01132 ↗First posted 31 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Understanding chart and table images is essential for applying vision-language models (VLMs) to real-world document understanding. While English benchmarks have advanced rapidly, non-English counterparts remain scarce, leaving it unclear whether this progress generalizes across languages. A key obstacle is the difficulty of collecting realistic and diverse non-English chart and table images at scale. To address this, we leverage governmental white papers as a source for benchmark construction, as they contain naturally occurring charts and tables across diverse formats and domains and are freely accessible in many countries. As a first instantiation, we introduce HakushoBench, a Japanese chart and table VQA benchmark built from 33 governmental white papers. HakushoBench contains 2,053 images spanning over 10 image types, with manually annotated and independently verified QA pairs designed to assess holistic understanding of charts and tables rather than local visual cues alone. Experiments across a broad range of VLMs show that HakushoBench is substantially harder than the existing Japanese benchmark and remains challenging for open-weight models: sub-10B open-weight models reach at most 58.6% accuracy, and even the flagship open-weight model Qwen3.5-397B-A17B trails Gemini~3~Pro by 8.1 points (85.8% vs. 93.9%), highlighting substantial room for improvement in complex chart and table understanding. We release our dataset and code.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 37 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

19 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answering16 Jul

Other references · 18

  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  7. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  8. 2025Humanity's Last Examcited by 491
  9. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  10. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  11. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  12. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
6 more
  1. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  2. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 108
  3. 2025A Rosetta Stone for AI Benchmarkscited by 36
  4. 2025MTabVQA: Evaluating Multi-Tabular Reasoning of Language Models in Visual Spacecited by 9
  5. 2025Evaluating Multimodal Large Language Models on Vertically Written Japanese Textcited by 4
  6. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Charts
  2. 2024MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding
  3. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts
  4. 2026JaWildText: A Benchmark for Vision-Language Models on Japanese Scene Text Understanding
  5. 2025MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding
  6. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs
  7. 2024Building and better understanding vision-language models: insights and future directions
  8. 2025InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information