# A Question-Answering Approach to Key Value Pair Extraction from Form-like Document Images

arXiv: https://arxiv.org/abs/2304.07957 · HTML: https://mlsanity.com/paper/2304.07957/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction](https://mlsanity.com/paper/2103.10213/) (2021)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021)
- [LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding](https://mlsanity.com/paper/2202.13669/) (2022)
- [BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents](https://mlsanity.com/paper/2108.04539/) (2021)
- [StrucTexT: Structured Text Understanding with Multi-Modal Transformers](https://mlsanity.com/paper/2108.02923/) (2021)
- [PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks](https://mlsanity.com/paper/2004.07464/) (2020)
- [XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understanding](https://mlsanity.com/paper/2203.06947/) (2022)
- [Spatial Dependency Parsing for Semi-Structured Document Information Extraction](https://mlsanity.com/paper/2005.00642/) (2020)
- [LAMBERT: Layout-Aware (Language) Modeling for information extraction](https://mlsanity.com/paper/2002.08087/) (2020)
- [DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding](https://mlsanity.com/paper/2010.11685/) (2020)
- [ViBERTgrid: A Jointly Trained Multi-Modal 2D Document Representation for Key Information Extraction from Documents](https://mlsanity.com/paper/2105.11672/) (2021)
- [Deep Visual Template-Free Form Parsing](https://mlsanity.com/paper/1909.02576/) (2019)
- [Entity Relation Extraction as Dependency Parsing in Visually Rich Documents](https://mlsanity.com/paper/2110.09915/) (2021)
- [Visual FUDGE: Form Understanding via Dynamic Graph Editing](https://mlsanity.com/paper/2105.08194/) (2021)
- [End-to-End Hierarchical Relation Extraction for Generic Form Understanding](https://mlsanity.com/paper/2106.00980/) (2021)
- [Value Retrieval with Arbitrary Queries for Form-like Documents](https://mlsanity.com/paper/2112.07820/) (2021)
- [End-to-End Information Extraction by Character-Level Embedding and Multi-Stage Attentional U-Net](https://mlsanity.com/paper/2106.00952/) (2021)
- [Text Classification Models for Form Entity Linking](https://mlsanity.com/paper/2112.07443/) (2021)

## Built on by (most cited first, 9 of 9)

- [Deep Learning based Visually Rich Document Content Understanding: A Survey](https://mlsanity.com/paper/2408.01287/) (2024)
- [DLAFormer: An End-to-End Transformer For Document Layout Analysis](https://mlsanity.com/paper/2405.11757/) (2024)
- [PPN: Parallel Pointer-based Network for Key Information Extraction with Complex Layouts](https://mlsanity.com/paper/2307.10551/) (2023)
- [AgenticIE: An Adaptive Agent for Information Extraction from Complex Regulatory Documents](https://mlsanity.com/paper/2509.11773/) (2025)
- [ProtoNER: Few shot Incremental Learning for Named Entity Recognition using Prototypical Networks](https://mlsanity.com/paper/2310.02372/) (2023)
- [PEneo: Unifying Line Extraction, Line Grouping, and Entity Linking for End-to-end Document Pair Extraction](https://mlsanity.com/paper/2401.03472/) (2024)
- [An agentic system with reinforcement-learned subsystem improvements for parsing form-like documents](https://mlsanity.com/paper/2505.13504/) (2025)
- [UniVIE: A Unified Label Space Approach to Visual Information Extraction from Form-like Documents](https://mlsanity.com/paper/2401.09220/) (2024)
- [Hierarchical Document Parsing via Large Margin Feature Matching and Heuristics](https://mlsanity.com/paper/2502.07442/) (2025)

## Abstract (arXiv)

In this paper, we present a new question-answering (QA) based key-value pair extraction approach, called KVPFormer, to robustly extracting key-value relationships between entities from form-like document images. Specifically, KVPFormer first identifies key entities from all entities in an image with a Transformer encoder, then takes these key entities as questions and feeds them into a Transformer decoder to predict their corresponding answers (i.e., value entities) in parallel. To achieve higher answer prediction accuracy, we propose a coarse-to-fine answer prediction approach further, which first extracts multiple answer candidates for each identified question in the coarse stage and then selects the most likely one among these candidates in the fine stage. In this way, the learning difficulty of answer prediction can be effectively reduced so that the prediction accuracy can be improved. Moreover, we introduce a spatial compatibility attention bias into the self-attention/cross-attention mechanism for Ours to better model the spatial interactions between entities. With these new techniques, our proposed Ours achieves state-of-the-art results on FUNSD and XFUND datasets, outperforming the previous best-performing method by 7.2% and 13.2% in F1 score, respectively.