# PEneo: Unifying Line Extraction, Line Grouping, and Entity Linking for End-to-end Document Pair Extraction

arXiv: https://arxiv.org/abs/2401.03472 · HTML: https://mlsanity.com/paper/2401.03472/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 7 Jan 2024.

## Builds on (most cited first, 28 of 28)

- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction](https://mlsanity.com/paper/2103.10213/) (2021)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [SelfDoc: Self-Supervised Document Representation Learning](https://mlsanity.com/paper/2106.03331/) (2021)
- [LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding](https://mlsanity.com/paper/2202.13669/) (2022)
- [BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents](https://mlsanity.com/paper/2108.04539/) (2021)
- [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://mlsanity.com/paper/2104.08836/) (2021)
- [StrucTexT: Structured Text Understanding with Multi-Modal Transformers](https://mlsanity.com/paper/2108.02923/) (2021)
- [Unified Pretraining Framework for Document Understanding](https://mlsanity.com/paper/2204.10939/) (2022)
- [End-to-end Document Recognition and Understanding with Dessurt](https://mlsanity.com/paper/2203.16618/) (2022)
- [TPLinker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linking](https://mlsanity.com/paper/2010.13415/) (2020)
- [XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understanding](https://mlsanity.com/paper/2203.06947/) (2022)
- [ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding](https://mlsanity.com/paper/2210.06155/) (2022)
- [Spatial Dependency Parsing for Semi-Structured Document Information Extraction](https://mlsanity.com/paper/2005.00642/) (2020)
- [Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solution](https://mlsanity.com/paper/2305.07498/) (2023)
- [Packed Levitated Marker for Entity and Relation Extraction](https://mlsanity.com/paper/2109.06067/) (2021)
- [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023)
- [GeoLayoutLM: Geometric Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2304.10759/) (2023)
- [Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction](https://mlsanity.com/paper/2310.11016/) (2023)
- [Modeling Entities as Semantic Points for Visual Information Extraction in the Wild](https://mlsanity.com/paper/2303.13095/) (2023)
- [DocTr: Document Transformer for Structured Information Extraction in Documents](https://mlsanity.com/paper/2307.07929/) (2023)
- [Visual FUDGE: Form Understanding via Dynamic Graph Editing](https://mlsanity.com/paper/2105.08194/) (2021)
- [A Question-Answering Approach to Key Value Pair Extraction from Form-like Document Images](https://mlsanity.com/paper/2304.07957/) (2023)

## Built on by (most cited first, 2 of 2)

- [Deep Learning based Key Information Extraction from Business Documents: Systematic Literature Review](https://mlsanity.com/paper/2408.06345/) (2024)
- [Unveiling the Deficiencies of Pre-trained Text-and-Layout Models in Real-world Visually-rich Document Information Extraction](https://mlsanity.com/paper/2402.02379/) (2024)

## Abstract (arXiv)

Document pair extraction aims to identify key and value entities as well as their relationships from visually-rich documents. Most existing methods divide it into two separate tasks: semantic entity recognition (SER) and relation extraction (RE). However, simply concatenating SER and RE serially can lead to severe error propagation, and it fails to handle cases like multi-line entities in real scenarios. To address these issues, this paper introduces a novel framework, PEneo (Pair Extraction new decoder option), which performs document pair extraction in a unified pipeline, incorporating three concurrent sub-tasks: line extraction, line grouping, and entity linking. This approach alleviates the error accumulation problem and can handle the case of multi-line entities. Furthermore, to better evaluate the model's performance and to facilitate future research on pair extraction, we introduce RFUND, a re-annotated version of the commonly used FUNSD and XFUND datasets, to make them more accurate and cover realistic situations. Experiments on various benchmarks demonstrate PEneo's superiority over previous pipelines, boosting the performance by a large margin (e.g., 19.89%-22.91% F1 score on RFUND-EN) when combined with various backbones like LiLT and LayoutLMv3, showing its effectiveness and generality. Codes and the new annotations are available at https://github.com/ZeningLin/PEneo.