mlsanityData to 14 Oct 2026

XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understanding

arXiv 2203.06947 ↗First posted 14 Mar 2022cs.CV cs.CLcited by 68

Not on the map. Why?

Abstract (arXiv)

Recently, various multimodal networks for Visually-Rich Document Understanding(VRDU) have been proposed, showing the promotion of transformers by integrating visual and layout information with the text embeddings. However, most existing approaches utilize the position embeddings to incorporate the sequence information, neglecting the noisy improper reading order obtained by OCR tools. In this paper, we propose a robust layout-aware multimodal network named XYLayoutLM to capture and leverage rich layout information from proper reading orders produced by our Augmented XY Cut. Moreover, a Dilated Conditional Position Encoding module is proposed to deal with the input sequence of variable lengths, and it additionally extracts local layout information from both textual and visual modalities while generating position embeddings. Experiment results show that our XYLayoutLM achieves competitive results on document understanding tasks.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 14 Mar 2022.

Builds on

23 of 35 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,133
  7. 2018Self-Attention with Relative Position Representationscited by 1,120
  8. 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 553
  9. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  10. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  11. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  12. 2019FUNSD: A Dataset for Form Understanding in Noisy Scanned Documentscited by 349
11 more
  1. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  2. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  3. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  4. 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
  5. 2021BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentscited by 104
  6. 2021LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingcited by 103
  7. 2018Chargrid: Towards Understanding 2D Documentscited by 100
  8. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  9. 2019Graph Convolution for Multimodal Information Extraction from Visually Rich Documentscited by 88
  10. 2021LayoutReader: Pre-training of Text and Layout for Reading Order Detectioncited by 80
  11. 2020LAMBERT: Layout-Aware (Language) Modeling for information extractioncited by 53

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 23; 2024: 18; 2025: 13; 2026: 4.

68 papers.

Sort:

By area:

  1. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  2. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  3. 2023KOSMOS-2.5: A Multimodal Literate Modelcited by 96
  4. 2024LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understandingcited by 95
  5. 2025MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigmcited by 87
  6. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  7. 2022ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understandingcited by 67
  8. 2023DocILE Benchmark for Document Information Localization and Extractioncited by 58
  9. 2023DocFormerv2: Local Features for Document Understandingcited by 46
  10. 2023GeoLayoutLM: Geometric Pre-training for Visual Information Extractioncited by 40
  11. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  12. 2024HRVDA: High-Resolution Visual Document Assistantcited by 37
38 more
  1. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  2. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  3. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  4. 2023Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answeringcited by 24
  5. 2023Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Predictioncited by 23
  6. 2025OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Modelscited by 23
  7. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
  8. 2022PP-StructureV2: A Stronger Document Analysis Systemcited by 20
  9. 2022Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understandingcited by 20
  10. 2023Modeling Entities as Semantic Points for Visual Information Extraction in the Wildcited by 20
  11. 2023FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extractioncited by 20
  12. 2023LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understandingcited by 17
  13. 2024Detect-Order-Construct: A Tree Construction based Approach for Hierarchical Document Structure Analysiscited by 15
  14. 2025Multimodal Tabular Reasoning with Privileged Structured Informationcited by 15
  15. 2023SwinDocSegmenter: An End-to-End Unified Domain Adaptive Transformer for Document Instance Segmentationcited by 13
  16. 2022XDoc: Unified Pre-training for Cross-Format Document Understandingcited by 12
  17. 2024Modeling Layout Reading Order as Ordering Relations for Visually-rich Document Understandingcited by 12
  18. 2024E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTioncited by 11
  19. 2022ERNIE-mmLayout: Multi-grained MultiModal Transformer for Document Understandingcited by 9
  20. 2023A Question-Answering Approach to Key Value Pair Extraction from Form-like Document Imagescited by 9
  21. 2025A Simple yet Effective Layout Token in Large Language Models for Document Understandingcited by 9
  22. 2025Benchmarking Graph Neural Networks for Document Layout Analysis in Public Affairscited by 7
  23. 2023Fast-StrucTexT: An Efficient Hourglass Transformer with Modality-guided Dynamic Token Merge for Document Understandingcited by 6
  24. 2023Do-GOOD: Towards Distribution Shift Evaluation for Pre-Trained Visual Document Understanding Modelscited by 6
  25. 2023Enhancing Visually-Rich Document Understanding via Layout Structure Modelingcited by 6
  26. 2024A Hybrid Approach for Document Layout Analysis in Document imagescited by 6
  27. 2024SAIL: Sample-Centric In-Context Learning for Document Information Extractioncited by 6
  28. 2023Text Reading Order in Uncontrolled Conditions by Sparse Graph Segmentationcited by 5
  29. 2025AI Agents and Agentic AI-Navigating a Plethora of Concepts for Future Manufacturingcited by 5
  30. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  31. 2024ReLayout: Towards Real-World Document Understanding via Layout-enhanced Pre-trainingcited by 4
  32. 2024Unveiling the Deficiencies of Pre-trained Text-and-Layout Models in Real-world Visually-rich Document Information Extractioncited by 3
  33. 2024DocMamba: Efficient Document Pre-training with State Space Modelcited by 3
  34. 2022Knowing Where and What: Unified Word Block Pretraining for Document Understandingcited by 2
  35. 2022Unimodal and Multimodal Representation Training for Relation Extractioncited by 2
  36. 2023Global Structure Knowledge-Guided Relation Extraction Method for Visually-Rich Documentcited by 2
  37. 2023DocTrack: A Visually-Rich Document Dataset Really Aligned with Human Eye Movement for Machine Readingcited by 2
  38. 2023A Scalable Framework for Table of Contents Extraction from Complex ESG Annual Reportscited by 2

50 most cited of 68 All 68 (JSON)