# Spatial Dual-Modality Graph Reasoning for Key Information Extraction

arXiv: https://arxiv.org/abs/2103.14470 · HTML: https://mlsanity.com/paper/2103.14470/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Understanding the Effective Receptive Field in Deep Convolutional Neural Networks](https://mlsanity.com/paper/1701.04128/) (2017)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [Videos as Space-Time Region Graphs](https://mlsanity.com/paper/1806.01810/) (2018)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [Discriminative Embeddings of Latent Variable Models for Structured Data](https://mlsanity.com/paper/1603.05629/) (2016)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [Cross-Sentence N-ary Relation Extraction with Graph LSTMs](https://mlsanity.com/paper/1708.03743/) (2017)
- [Chargrid: Towards Understanding 2D Documents](https://mlsanity.com/paper/1809.08799/) (2018)
- [Graph Convolution for Multimodal Information Extraction from Visually Rich Documents](https://mlsanity.com/paper/1903.11279/) (2019)
- [BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection](https://mlsanity.com/paper/1902.00038/) (2019)
- [VAIN: Attentional Multi-agent Predictive Modeling](https://mlsanity.com/paper/1706.06122/) (2017)
- [Semantic Object Parsing with Graph LSTM](https://mlsanity.com/paper/1603.07063/) (2016)
- [Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition](https://mlsanity.com/paper/1712.07465/) (2017)
- [Deep Reasoning with Knowledge Graph for Social Relationship Understanding](https://mlsanity.com/paper/1807.00504/) (2018)
- [CloudScan - A configuration-free invoice analysis system using recurrent neural networks](https://mlsanity.com/paper/1708.07403/) (2017)
- [Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition](https://mlsanity.com/paper/1712.05134/) (2017)
- [Fashion Retrieval via Graph Reasoning Networks on a Similarity Pyramid](https://mlsanity.com/paper/1908.11754/) (2019)

## Built on by (most cited first, 30 of 43)

- [Detect Anything via Next Point Prediction](https://mlsanity.com/paper/2510.12798/) (2025)
- [DocILE Benchmark for Document Information Localization and Extraction](https://mlsanity.com/paper/2302.05658/) (2023)
- [Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solution](https://mlsanity.com/paper/2305.07498/) (2023)
- [MMOCR: A Comprehensive Toolbox for Text Detection, Recognition and Understanding](https://mlsanity.com/paper/2108.06543/) (2021)
- [LongCat-Next: Lexicalizing Modalities as Discrete Tokens](https://mlsanity.com/paper/2603.27538/) (2026)
- [DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming](https://mlsanity.com/paper/2406.19101/) (2024)
- [InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions](https://mlsanity.com/paper/2401.13313/) (2024)
- [BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices](https://mlsanity.com/paper/2411.10640/) (2024)
- [OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning](https://mlsanity.com/paper/2505.17163/) (2025)
- [DocParser: End-to-end OCR-free Information Extraction from Visually Rich Documents](https://mlsanity.com/paper/2304.12484/) (2023)
- [MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding](https://mlsanity.com/paper/2410.21311/) (2024)
- [Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models](https://mlsanity.com/paper/2506.20168/) (2025)
- [Business Document Information Extraction: Towards Practical Benchmarks](https://mlsanity.com/paper/2206.11229/) (2022)
- [StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyond](https://mlsanity.com/paper/2405.21013/) (2024)
- [FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding](https://mlsanity.com/paper/2505.17330/) (2025)
- [Universal Graph Continual Learning](https://mlsanity.com/paper/2308.13982/) (2023)
- [Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report](https://mlsanity.com/paper/2406.11403/) (2024)
- [Vision as Unified Multimodal Generation](https://mlsanity.com/paper/2607.06560/) (2026)
- [TRIE++: Towards End-to-End Information Extraction from Visually Rich Documents](https://mlsanity.com/paper/2207.06744/) (2022)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [AIForge-Doc: A Benchmark for Detecting AI-Forged Tampering in Financial and Form Documents](https://mlsanity.com/paper/2602.20569/) (2026)
- ["What is the value of {templates}?" Rethinking Document Information Extraction Datasets for LLMs](https://mlsanity.com/paper/2410.15484/) (2024)
- [Unveiling the Deficiencies of Pre-trained Text-and-Layout Models in Real-world Visually-rich Document Information Extraction](https://mlsanity.com/paper/2402.02379/) (2024)
- [Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding](https://mlsanity.com/paper/2505.05446/) (2025)
- [Toward Real-world Text Image Forgery Localization: Structured and Interpretable Data Synthesis](https://mlsanity.com/paper/2511.12658/) (2025)
- [When the Forger Is the Judge: GPT-Image-2 Cannot Recognize Its Own Faked Documents](https://mlsanity.com/paper/2604.25213/) (2026)
- [PrIeD-KIE: Towards Privacy Preserved Document Key Information Extraction](https://mlsanity.com/paper/2310.03777/) (2023)
- [ChatGPT Images 2.5 on Forgery Tasks: Testing Advertised Improvements Against Known Answers](https://mlsanity.com/paper/2609.13617/) (2026)
- [DavarOCR: A Toolbox for OCR and Multi-Modal Document Understanding](https://mlsanity.com/paper/2207.06695/) (2022)
- [DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiency](https://mlsanity.com/paper/2311.05778/) (2023)

## Abstract (arXiv)

Key information extraction from document images is of paramount importance in office automation. Conventional template matching based approaches fail to generalize well to document images of unseen templates, and are not robust against text recognition errors. In this paper, we propose an end-to-end Spatial Dual-Modality Graph Reasoning method (SDMG-R) to extract key information from unstructured document images. We model document images as dual-modality graphs, nodes of which encode both the visual and textual features of detected text regions, and edges of which represent the spatial relations between neighboring text regions. The key information extraction is solved by iteratively propagating messages along graph edges and reasoning the categories of graph nodes. In order to roundly evaluate our proposed method as well as boost the future research, we release a new dataset named WildReceipt, which is collected and annotated tailored for the evaluation of key information extraction from document images of unseen templates in the wild. It contains 25 key information categories, a total of about 69000 text boxes, and is about 2 times larger than the existing public datasets. Extensive experiments validate that all information including visual features, textual features and spatial relations can benefit key information extraction. It has been shown that SDMG-R can effectively extract key information from document images of unseen templates, and obtain new state-of-the-art results on the recent popular benchmark SROIE and our WildReceipt. Our code and dataset will be publicly released.