# FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction

arXiv: https://arxiv.org/abs/2305.02549 · HTML: https://mlsanity.com/paper/2305.02549/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 71 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents](https://mlsanity.com/paper/1905.13538/) (2019)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction](https://mlsanity.com/paper/2103.10213/) (2021)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Graph Matching Networks for Learning the Similarity of Graph Structured Objects](https://mlsanity.com/paper/1904.12787/) (2019)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [Representing Long-Range Context for Graph Neural Networks with Global Attention](https://mlsanity.com/paper/2201.08821/) (2022)
- [DiT: Self-supervised Pre-training for Document Image Transformer](https://mlsanity.com/paper/2203.02378/) (2022)
- [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021)
- [SelfDoc: Self-Supervised Document Representation Learning](https://mlsanity.com/paper/2106.03331/) (2021)
- [An Empirical Study of Graph Contrastive Learning](https://mlsanity.com/paper/2109.01116/) (2021)

## Built on by (most cited first, 20 of 20)

- [LMDX: Language Model-based Document Information Extraction and Localization](https://mlsanity.com/paper/2309.10952/) (2023)
- [Graph Transformers: A Survey](https://mlsanity.com/paper/2407.09777/) (2024)
- [Learning on Multimodal Graphs: A Survey](https://mlsanity.com/paper/2402.05322/) (2024)
- [Deep Learning based Visually Rich Document Content Understanding: A Survey](https://mlsanity.com/paper/2408.01287/) (2024)
- [Visually Guided Generative Text-Layout Pre-training for Document Intelligence](https://mlsanity.com/paper/2403.16516/) (2024)
- [Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments](https://mlsanity.com/paper/2505.13535/) (2025)
- [BuDDIE: A Business Document Dataset for Multi-task Information Extraction](https://mlsanity.com/paper/2404.04003/) (2024)
- [KVP10k : A Comprehensive Dataset for Key-Value Pair Extraction in Business Documents](https://mlsanity.com/paper/2405.00505/) (2024)
- [3MVRD: Multimodal Multi-task Multi-teacher Visually-Rich Form Document Understanding](https://mlsanity.com/paper/2402.17983/) (2024)
- [Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulation](https://mlsanity.com/paper/2305.14828/) (2023)
- [A Scalable Framework for Table of Contents Extraction from Complex ESG Annual Reports](https://mlsanity.com/paper/2310.18073/) (2023)
- [EMC2: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence](https://mlsanity.com/paper/2404.10575/) (2024)
- [An agentic system with reinforcement-learned subsystem improvements for parsing form-like documents](https://mlsanity.com/paper/2505.13504/) (2025)
- [RE2: Region-Aware Relation Extraction from Visually Rich Documents](https://mlsanity.com/paper/2305.14590/) (2023)
- [Towards Human-Like Machine Comprehension: Few-Shot Relational Learning in Visually-Rich Documents](https://mlsanity.com/paper/2403.15765/) (2024)
- [HIP: Hierarchical Point Modeling and Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2411.01139/) (2024)
- [Information Extraction from Heterogeneous Documents without Ground Truth Labels using Synthetic Label Generation and Knowledge Distillation](https://mlsanity.com/paper/2411.14957/) (2024)
- [R2-CoD: Understanding Text-Graph Complementarity in Relational Reasoning via Knowledge Co-Distillation](https://mlsanity.com/paper/2508.01475/) (2025)
- [LAVA: Logic-Aware Validation and Augmentation Framework for Large-Scale Financial Document Auditing](https://mlsanity.com/paper/2608.16763/) (2026)
- [Mind the Gaps: A Curated Benchmark for Form Field Detection](https://mlsanity.com/paper/2609.23679/) (2026)

## Abstract (arXiv)

The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.