mlsanityData to 7 Oct 2026

FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extraction

arXiv 2203.08411 ↗First posted 16 Mar 2022cs.CL cs.CV cs.LGcited by 64

Not on the map. Why?

Abstract (arXiv)

Sequence modeling has demonstrated state-of-the-art performance on natural language and document understanding tasks. However, it is challenging to correctly serialize tokens in form-like documents in practice due to their variety of layout patterns. We propose FormNet, a structure-aware sequence model to mitigate the suboptimal serialization of forms. First, we design Rich Attention that leverages the spatial relationship between tokens in a form for more precise attention score calculation. Second, we construct Super-Tokens for each word by embedding representations from their neighboring tokens through graph convolutions. FormNet therefore explicitly recovers local syntactic information that may have been lost during serialization. In experiments, FormNet outperforms existing methods with a more compact model size and less pre-training data, establishing new state-of-the-art performance on CORD, FUNSD and Payment benchmarks.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 37 of its 59 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

37 of 59 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2018Self-Attention with Relative Position Representationscited by 1,112
  9. 2018Image Transformercited by 770
  10. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
  11. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
  12. 2019Is Attention Interpretable?cited by 387
25 more
  1. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 345
  2. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  3. 2021ICDAR2019 Competition on Scanned Receipt OCR and Information Extractioncited by 254
  4. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
  5. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  6. 2021Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformercited by 134
  7. 2017Cross-Sentence N-ary Relation Extraction with Graph LSTMscited by 101
  8. 2018Chargrid: Towards Understanding 2D Documentscited by 100
  9. 2017Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Networkcited by 92
  10. 2019Graph Convolution for Multimodal Information Extraction from Visually Rich Documentscited by 88
  11. 2020Deep Relational Reasoning Graph Network for Arbitrary Shape Text Detectioncited by 77
  12. 2020PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networkscited by 70
  13. 2020Spatial Dependency Parsing for Semi-Structured Document Information Extractioncited by 65
  14. 2019BERTgrid: Contextualized Embedding for 2D Document Representation and Understandingcited by 63
  15. 2020LAMBERT: Layout-Aware (Language) Modeling for information extractioncited by 53
  16. 2018GraphIE: A Graph-Based Framework for Information Extractioncited by 45
  17. 2020ETC: Encoding Long and Structured Inputs in Transformerscited by 34
  18. 2017CloudScan - A configuration-free invoice analysis system using recurrent neural networkscited by 33
  19. 2020Kleister: A novel task for Information Extraction involving Long Documents with Complex Layoutcited by 27
  20. 2018N-ary Relation Extraction using Graph State LSTMcited by 26
  21. 2019Deep Visual Template-Free Form Parsingcited by 24
  22. 2021ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extractioncited by 22
  23. 2019CUTIE: Learning to Understand Documents with Convolutional Universal Text Information Extractorcited by 21
  24. 2021Post-OCR Paragraph Recognition by Graph Convolutional Networkscited by 16
  25. 2021Form2Seq : A Framework for Higher-Order Form Structure Extractioncited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 23; 2024: 15; 2025: 8; 2026: 7.

64 papers.

Sort:

By area:

  1. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  2. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  3. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  4. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  5. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  6. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  7. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  8. 2023LMDX: Language Model-based Document Information Extraction and Localizationcited by 37
  9. 2022VRDU: A Benchmark for Visually-rich Document Understandingcited by 35
  10. 2023Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answeringcited by 24
  11. 2022Multimodal Pre-training Based on Graph Attention Network for Document Understandingcited by 23
  12. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 22
38 more
  1. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  2. 2023FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extractioncited by 20
  3. 2023DocTr: Document Transformer for Structured Information Extraction in Documentscited by 18
  4. 2024Visually Guided Generative Text-Layout Pre-training for Document Intelligencecited by 12
  5. 2024Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understandingcited by 12
  6. 2025FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understandingcited by 10
  7. 2022ERNIE-mmLayout: Multi-grained MultiModal Transformer for Document Understandingcited by 9
  8. 2023Hierarchical Text Spotter for Joint Text Spotting and Layout Analysiscited by 8
  9. 2022QueryForm: A Simple Zero-shot Form Entity Query Frameworkcited by 7
  10. 2023Analyzing the Efficacy of an LLM-Only Approach for Image-based Document Question Answeringcited by 7
  11. 2022Test-Time Adaptation for Visual Document Understandingcited by 6
  12. 2023Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided Dynamic Token Merge for Document Understandingcited by 6
  13. 2023Enhancing Visually-Rich Document Understanding via Layout Structure Modelingcited by 6
  14. 2024DocGraphLM: Documental Graph Language Model for Information Extractioncited by 6
  15. 2024Deep Learning based Key Information Extraction from Business Documents: Systematic Literature Reviewcited by 6
  16. 2025Problem Solved? Information Extraction Design Space for Layout-Rich Documents using LLMscited by 6
  17. 2023Text Reading Order in Uncontrolled Conditions by Sparse Graph Segmentationcited by 5
  18. 2025Hybrid OCR-LLM Framework for Enterprise-Scale Document Information Extraction Under Copy-heavy Taskcited by 5
  19. 2023DocumentNet: Bridging the Data Gap in Document Pre-Trainingcited by 4
  20. 2024Hypergraph based Understanding for Document Semantic Entity Recognitioncited by 4
  21. 2024ReLayout: Towards Real-World Document Understanding via Layout-enhanced Pre-trainingcited by 4
  22. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  23. 2026UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documentscited by 4
  24. 2022Knowing Where and What: Unified Word Block Pretraining for Document Understandingcited by 2
  25. 2023Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulationcited by 2
  26. 2023ProtoNER: Few shot Incremental Learning for Named Entity Recognition using Prototypical Networkscited by 2
  27. 2023PrIeD-KIE: Towards Privacy Preserved Document Key Information Extractioncited by 2
  28. 2023A Scalable Framework for Table of Contents Extraction from Complex ESG Annual Reportscited by 2
  29. 2023On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrievalcited by 2
  30. 2024Watermark Text Pattern Spotting in Document Imagescited by 2
  31. 2024Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibitscited by 2
  32. 2025FlexDoc: Parameterized Sampling for Diverse Multilingual Synthetic Documents for Training Document Understanding Modelscited by 2
  33. 2022Alignment-Enriched Tuning for Patch-Level Pre-trained Document Image Modelscited by 1
  34. 2023GVdoc: Graph-based Visual Document Classificationcited by 1
  35. 2023GlobalDoc: A Cross-Modal Vision-Language Framework for Real-World Document Image Retrieval and Classificationcited by 1
  36. 2023Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learningcited by 1
  37. 2025FormGym: Doing Paperwork with Agentscited by 1
  38. 2025DP-DocLDM: Differentially Private Document Image Generation using Latent Diffusion Modelscited by 1

50 most cited of 64 All 64 (JSON)