mlsanityData to 7 Oct 2026

Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network

arXiv 1706.02337 ↗First posted 7 Jun 2017cs.CV cs.LGcited by 92

Not on the map. Why?

Abstract (arXiv)

We present an end-to-end, multimodal, fully convolutional network for extracting semantic structures from document images. We consider document semantic structure extraction as a pixel-wise segmentation task, and propose a unified model that classifies pixels based not only on their visual appearance, as in the traditional page segmentation task, but also on the content of underlying text. Moreover, we propose an efficient synthetic document generation process that we use to generate pretraining data for our network. Once the network is trained on a large set of synthetic documents, we fine-tune the network on unlabeled real documents using a semi-supervised approach. We systematically study the optimum network architecture and show that both our multimodal approach and the synthetic data pretraining significantly boost the performance.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 24; 2023: 16; 2024: 11; 2025: 6; 2026: 2.

92 papers.

Sort:

By area:

  1. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  2. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  3. 2022LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingcited by 107
  4. 2022Towards End-to-End Unified Scene Text Detection and Layout Analysiscited by 96
  5. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  6. 2021Variational Transformer Networks for Layout Generationcited by 88
  7. 2022GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversationcited by 86
  8. 2024DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perceptioncited by 80
  9. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  10. 2022FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extractioncited by 64
  11. 2022DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognitioncited by 62
  12. 2021Document AI: Benchmarks, Models and Applicationscited by 61
38 more
  1. 2023Vision Grid Transformer for Document Layout Analysiscited by 45
  2. 2023M6Doc: A Large-Scale Multi-Format, Multi-Type, Multi-Layout, Multi-Language, Multi-Annotation Category Dataset for Modern Document Layout Analysiscited by 44
  3. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  4. 2021Split, embed and merge: An accurate table structure recognizercited by 42
  5. 2021VSR: A Unified Framework for Document Layout Analysis combining Vision, Semantics and Relationscited by 40
  6. 2022WebFormer: The Web-page Transformer for Structure Information Extractioncited by 34
  7. 2022Robust Table Detection and Structure Recognition from Heterogeneous Document Imagescited by 30
  8. 2021ViBERTgrid: A Jointly Trained Multi-Modal 2D Document Representation for Key Information Extraction from Documentscited by 29
  9. 2021TPSNet: Reverse Thinking of Thin Plate Splines for Arbitrary Shape Scene Text Representationcited by 26
  10. 2023Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answeringcited by 24
  11. 2024HTR-VT: Handwritten Text Recognition with Vision Transformercited by 24
  12. 2022Multimodal Pre-training Based on Graph Attention Network for Document Understandingcited by 23
  13. 2023ICDAR 2023 Competition on Hierarchical Text Detection and Recognitioncited by 23
  14. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  15. 2021Page Layout Analysis System for Unconstrained Historic Documentscited by 20
  16. 2022Bi-VLDoc: Bidirectional Vision-Language Modeling for Visually-Rich Document Understandingcited by 18
  17. 2023DocTr: Document Transformer for Structured Information Extraction in Documentscited by 18
  18. 2023LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understandingcited by 17
  19. 2021Post-OCR Paragraph Recognition by Graph Convolutional Networkscited by 16
  20. 2021Form2Seq : A Framework for Higher-Order Form Structure Extractioncited by 16
  21. 2021LAMPRET: Layout-Aware Multimodal PreTraining for Document Understandingcited by 15
  22. 2021Visual FUDGE: Form Understanding via Dynamic Graph Editingcited by 15
  23. 2022VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classificationcited by 14
  24. 2024Detect-Order-Construct: A Tree Construction based Approach for Hierarchical Document Structure Analysiscited by 14
  25. 2024RoDLA: Benchmarking the Robustness of Document Layout Analysis Modelscited by 12
  26. 2022SALAD: Source-free Active Label-Agnostic Domain Adaptation for Classification, Segmentation and Detectioncited by 10
  27. 2022ERNIE-mmLayout: Multi-grained MultiModal Transformer for Document Understandingcited by 9
  28. 2023Hierarchical Text Spotter for Joint Text Spotting and Layout Analysiscited by 8
  29. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  30. 2022Unified Line and Paragraph Detection by Graph Convolutional Networkscited by 7
  31. 2022Page Layout Analysis of Text-heavy Historical Documents: a Comparison of Textual and Visual Approachescited by 7
  32. 2023Towards End-to-End Semi-Supervised Table Detection with Deformable Transformercited by 7
  33. 2023Bridging the Performance Gap between DETR and R-CNN for Graphical Object Detection in Document Imagescited by 7
  34. 2021Multi-Modal Association based Grouping for Form Structure Extractioncited by 6
  35. 2022TRIE++: Towards End-to-End Information Extraction from Visually Rich Documentscited by 6
  36. 2023TRACE: Table Reconstruction Aligned to Corner and Edgescited by 6
  37. 2025Visual Template Inference for Data Extraction from Documentscited by 6
  38. 2021Magic Layouts: Structural Prior for Component Detection in User Interface Designscited by 5

50 most cited of 92 All 92 (JSON)