# DuwatBench: Bridging Language and Visual Heritage through an Arabic Calligraphy Benchmark for Multimodal Understanding

arXiv: https://arxiv.org/abs/2601.19898 · HTML: https://mlsanity.com/paper/2601.19898/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 10 of 10)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [PalmX 2025: The First Shared Task on Benchmarking LLMs on Arabic and Islamic Culture](https://mlsanity.com/paper/2509.02550/) (2025)
- [CAMEL-Bench: A Comprehensive Arabic LMM Benchmark](https://mlsanity.com/paper/2410.18976/) (2024)
- [Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic](https://mlsanity.com/paper/2407.18129/) (2024)
- [Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions](https://mlsanity.com/paper/2508.08287/) (2025)
- [AIN: The Arabic INclusive Large Multimodal Model](https://mlsanity.com/paper/2502.00094/) (2025)
- [QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation](https://mlsanity.com/paper/2506.02295/) (2025)
- [ARB: A Comprehensive Arabic Multimodal Reasoning Benchmark](https://mlsanity.com/paper/2505.17021/) (2025)
- [Calliar: An Online Handwritten Dataset for Arabic Calligraphy](https://mlsanity.com/paper/2106.10745/) (2021)

## Abstract (arXiv)

Arabic calligraphy represents one of the richest visual traditions of the Arabic language, blending linguistic meaning with artistic form. Although multimodal models have advanced across languages, their ability to process Arabic script, especially in artistic and stylized calligraphic forms, remains largely unexplored. To address this gap, we present DuwatBench, a benchmark of 1,272 curated samples containing about 1,475 unique words across six classical and modern calligraphic styles, each paired with sentence-level detection annotations. The dataset reflects real-world challenges in Arabic writing, such as complex stroke patterns, dense ligatures, and stylistic variations that often challenge standard text recognition systems. Using DuwatBench, we evaluated 13 leading Arabic and multilingual multimodal models and showed that while they perform well on clean text, they struggle with calligraphic variation, artistic distortions, and precise visual-text alignment. By publicly releasing DuwatBench and its annotations, we aim to advance culturally grounded multimodal research, foster fair inclusion of the Arabic language and visual heritage in AI systems, and support continued progress in this area. Our dataset (https://huggingface.co/datasets/MBZUAI/DuwatBench) and evaluation suit (https://github.com/mbzuai-oryx/DuwatBench) are publicly available.