# Robustness Evaluation of OCR-based Visual Document Understanding under Multi-Modal Adversarial Attacks

arXiv: https://arxiv.org/abs/2506.16407 · HTML: https://mlsanity.com/paper/2506.16407/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Builds on (most cited first, 14 of 14)

- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction](https://mlsanity.com/paper/2103.10213/) (2021)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [ImgTrojan: Jailbreaking Vision-Language Models with ONE Image](https://mlsanity.com/paper/2403.02910/) (2024)
- [GeoLayoutLM: Geometric Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2304.10759/) (2023)
- [RoDLA: Benchmarking the Robustness of Document Layout Analysis Models](https://mlsanity.com/paper/2403.14442/) (2024)
- [Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Models](https://mlsanity.com/paper/2306.02623/) (2023)
- [Adversarial Bounding Boxes Generation (ABBG) Attack against Visual Object Trackers](https://mlsanity.com/paper/2411.17468/) (2024)
- [Distortion-Aware Adversarial Attacks on Bounding Boxes of Object Detectors](https://mlsanity.com/paper/2412.18815/) (2024)

## Built on by (most cited first, 1 of 1)

- [Search-Based Risk Feature Discovery in Document Structure Spaces under a Constrained Budget](https://mlsanity.com/paper/2601.21608/) (2026)

## Abstract (arXiv)

Visual Document Understanding (VDU) systems have achieved strong performance in information extraction by integrating textual, layout, and visual signals. However, their robustness under realistic adversarial perturbations remains insufficiently explored. We introduce the first unified framework for generating and evaluating multi-modal adversarial attacks on OCR-based VDU models. Our method covers six gradient-based layout attack scenarios, incorporating manipulations of OCR bounding boxes, pixels, and texts across both word and line granularities, with constraints on layout perturbation budget (e.g., IoU >= 0.6) to preserve plausibility. Experimental results across four datasets (FUNSD, CORD, SROIE, DocVQA) and six model families demonstrate that line-level attacks and compound perturbations (BBox + Pixel + Text) yield the most severe performance degradation. Projected Gradient Descent (PGD)-based BBox perturbations outperform random-shift baselines in all investigated models. Ablation studies further validate the impact of layout budget, text modification, and adversarial transferability.