# Linguistically Informed Multimodal Fusion for Vietnamese Scene-Text Image Captioning: Dataset, Graph Framework, and Phonological Attention

arXiv: https://arxiv.org/abs/2604.27712 · HTML: https://mlsanity.com/paper/2604.27712/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 62 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 2 of 2)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Towards Signboard-Oriented Visual Question Answering: ViSignVQA Dataset, Method and Benchmark](https://mlsanity.com/paper/2512.22218/) (2025)

## Abstract (arXiv)

Scene-text image captioning requires fusing three information streams -- visual features, OCR-detected text, and linguistic knowledge -- to generate descriptions that faithfully integrate text visible in images. Existing fusion approaches treat text as language-agnostic, which fails for Vietnamese: a tonal language where diacritics alter word meaning, OCR errors are pervasive, and word boundaries are ambiguous. We argue that Vietnamese scene-text captioning demands textitlinguistically informed multimodal fusion, where language-specific structural knowledge is explicitly incorporated into the fusion mechanism. Motivated from these insights, we propose HSTFG (Heterogeneous Scene-Text Fusion Graph), a general-purpose graph fusion framework with learned spatial attention bias, and show through topology analysis that cross-modal graph edges are harmful for scene-text fusion. Building on this finding, we design PhonoSTFG (Phonological Scene-Text Fusion Graph) which specializes graph-level fusion for Vietnamese linguistic reasoning. To support evaluation, we introduce ViTextCaps, the first large-scale Vietnamese scene-text captioning dataset (textbf15,729 images with textbf74,970 captions), with comprehensive linguistic analysis showing that 52.8% of the vocabulary is at risk of diacritic collision.