# Notes on Applicability of GPT-4 to Document Understanding

arXiv: https://arxiv.org/abs/2405.18433 · HTML: https://mlsanity.com/paper/2405.18433/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 10 of 10)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions](https://mlsanity.com/paper/2401.13313/) (2024)
- [GRAM: Global Reasoning for Multi-Page VQA](https://mlsanity.com/paper/2401.03411/) (2024)
- [Document Visual Question Answering Challenge 2020](https://mlsanity.com/paper/2008.08899/) (2020)
- [From Dataset Recycling to Multi-Property Extraction and Beyond](https://mlsanity.com/paper/2011.03228/) (2020)

## Built on by (most cited first, 7 of 7)

- [Improve Vision Language Model Chain-of-thought Reasoning](https://mlsanity.com/paper/2410.16198/) (2024)
- [Large Language Models for Page Stream Segmentation](https://mlsanity.com/paper/2408.11981/) (2024)
- [The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formatting](https://mlsanity.com/paper/2505.12837/) (2025)
- [Arctic-TILT. Business Document Understanding at Sub-Billion Scale](https://mlsanity.com/paper/2408.04632/) (2024)
- [Entropy Heat-Mapping: Localizing GPT-Based OCR Errors with Sliding-Window Shannon Analysis](https://mlsanity.com/paper/2505.00746/) (2025)
- [Hespi: A pipeline for automatically detecting information from hebarium specimen sheets](https://mlsanity.com/paper/2410.08740/) (2024)
- [Unchecked and Overlooked: Addressing the Checkbox Blind Spot in Large Language Models with CheckboxQA](https://mlsanity.com/paper/2504.10419/) (2025)

## Abstract (arXiv)

We perform a missing, reproducible evaluation of all publicly available GPT-4 family models concerning the Document Understanding field, where it is frequently required to comprehend text spacial arrangement and visual clues in addition to textual semantics. Benchmark results indicate that though it is hard to achieve satisfactory results with text-only models, GPT-4 Vision Turbo performs well when one provides both text recognized by an external OCR engine and document images on the input. Evaluation is followed by analyses that suggest possible contamination of textual GPT-4 models and indicate the significant performance drop for lengthy documents.