# Cost-effective End-to-end Information Extraction for Semi-structured Document Images

arXiv: https://arxiv.org/abs/2104.08041 · HTML: https://mlsanity.com/paper/2104.08041/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [Image-based table recognition: data, model, and evaluation](https://mlsanity.com/paper/1911.10683/) (2019)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Coarse-to-Fine Decoding for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04793/) (2018)
- [Recurrent Neural Network Grammars](https://mlsanity.com/paper/1602.07776/) (2016)
- [Chargrid: Towards Understanding 2D Documents](https://mlsanity.com/paper/1809.08799/) (2018)
- [TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation](https://mlsanity.com/paper/1810.02720/) (2018)
- [Graph Convolution for Multimodal Information Extraction from Visually Rich Documents](https://mlsanity.com/paper/1903.11279/) (2019)
- [PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks](https://mlsanity.com/paper/2004.07464/) (2020)
- [Spatial Dependency Parsing for Semi-Structured Document Information Extraction](https://mlsanity.com/paper/2005.00642/) (2020)
- [BERTgrid: Contextualized Embedding for 2D Document Representation and Understanding](https://mlsanity.com/paper/1909.04948/) (2019)
- [GraphIE: A Graph-Based Framework for Information Extraction](https://mlsanity.com/paper/1810.13083/) (2018)
- [Transition-based Parsing with Stack-Transformers](https://mlsanity.com/paper/2010.10669/) (2020)
- [CloudScan - A configuration-free invoice analysis system using recurrent neural networks](https://mlsanity.com/paper/1708.07403/) (2017)
- [CUTIE: Learning to Understand Documents with Convolutional Universal Text Information Extractor](https://mlsanity.com/paper/1903.12363/) (2019)

## Built on by (most cited first, 14 of 14)

- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [End-to-end Document Recognition and Understanding with Dessurt](https://mlsanity.com/paper/2203.16618/) (2022)
- [A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction](https://mlsanity.com/paper/2206.05224/) (2022)
- [VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document Understanding](https://mlsanity.com/paper/2407.12594/) (2024)
- [ReALM: Reference Resolution As Language Modeling](https://mlsanity.com/paper/2403.20329/) (2024)
- [On Web-based Visual Corpus Construction for Visual Document Understanding](https://mlsanity.com/paper/2211.03256/) (2022)
- [Data-efficient End-to-end Information Extraction for Statistical Legal Analysis](https://mlsanity.com/paper/2211.01692/) (2022)
- [TreeForm: End-to-end Annotation and Evaluation for Form Document Parsing](https://mlsanity.com/paper/2402.05282/) (2024)
- [Sequence-to-Sequence Pre-training with Unified Modality Masking for Visual Document Understanding](https://mlsanity.com/paper/2305.10448/) (2023)
- [Enhancing Large Multimodal Models in Key Information Extraction via Scene-Aware Document Synthesis](https://mlsanity.com/paper/2607.04636/) (2026)
- [Semi-Structured Query Grounding for Document-Oriented Databases with Deep Retrieval and Its Application to Receipt and POI Matching](https://mlsanity.com/paper/2202.13959/) (2022)
- [DUBLIN -- Document Understanding By Language-Image Network](https://mlsanity.com/paper/2305.14218/) (2023)
- [Noise-Aware Training of Layout-Aware Language Models](https://mlsanity.com/paper/2404.00488/) (2024)
- [Neurosymbolic Information Extraction from Transactional Documents](https://mlsanity.com/paper/2512.09666/) (2025)

## Abstract (arXiv)

A real-world information extraction (IE) system for semi-structured document images often involves a long pipeline of multiple modules, whose complexity dramatically increases its development and maintenance cost. One can instead consider an end-to-end model that directly maps the input to the target output and simplify the entire process. However, such generation approach is known to lead to unstable performance if not designed carefully. Here we present our recent effort on transitioning from our existing pipeline-based IE system to an end-to-end system focusing on practical challenges that are associated with replacing and deploying the system in real, large-scale production. By carefully formulating document IE as a sequence generation task, we show that a single end-to-end IE system can be built and still achieve competent performance.