# Joint Extraction Matters: Prompt-Based Visual Question Answering for Multi-Field Document Information Extraction

arXiv: https://arxiv.org/abs/2503.16868 · HTML: https://mlsanity.com/paper/2503.16868/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction](https://mlsanity.com/paper/2103.10213/) (2021)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [VisualMRC: Machine Reading Comprehension on Document Images](https://mlsanity.com/paper/2101.11272/) (2021)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [LaTr: Layout-Aware Transformer for Scene-Text VQA](https://mlsanity.com/paper/2112.12494/) (2021)
- [DocFormerv2: Local Features for Document Understanding](https://mlsanity.com/paper/2306.01733/) (2023)
- [IDPG: An Instance-Dependent Prompt Generation Method](https://mlsanity.com/paper/2204.04497/) (2022)
- [Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods](https://mlsanity.com/paper/2305.18156/) (2023)
- [V-Doc : Visual questions answers with Documents](https://mlsanity.com/paper/2205.13724/) (2022)
- [Dynamic Prompting: A Unified Framework for Prompt Tuning](https://mlsanity.com/paper/2303.02909/) (2023)
- [Multiple-Question Multiple-Answer Text-VQA](https://mlsanity.com/paper/2311.08622/) (2023)

## Abstract (arXiv)

Visual question answering (VQA) has emerged as a flexible approach for extracting specific pieces of information from document images. However, existing work typically queries each field in isolation, overlooking potential dependencies across multiple items. This paper investigates the merits of extracting multiple fields jointly versus separately. Through experiments on multiple large vision language models and datasets, we show that jointly extracting fields often improves accuracy, especially when the fields share strong numeric or contextual dependencies. We further analyze how performance scales with the number of requested items and use a regression based metric to quantify inter field relationships. Our results suggest that multi field prompts can mitigate confusion arising from similar surface forms and related numeric values, providing practical methods for designing robust VQA systems in document information extraction tasks.