# VSR: A Unified Framework for Document Layout Analysis combining Vision, Semantics and Relations

arXiv: https://arxiv.org/abs/2105.06220 · HTML: https://mlsanity.com/paper/2105.06220/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 13 May 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [DocBank: A Benchmark Dataset for Document Layout Analysis](https://mlsanity.com/paper/2006.01038/) (2020)
- [Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network](https://mlsanity.com/paper/1706.02337/) (2017)
- [Graph Convolution for Multimodal Information Extraction from Visually Rich Documents](https://mlsanity.com/paper/1903.11279/) (2019)
- [PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks](https://mlsanity.com/paper/2004.07464/) (2020)
- [TRIE: End-to-End Text Reading and Information Extraction for Document Understanding](https://mlsanity.com/paper/2005.13118/) (2020)
- [Extracting Scientific Figures with Distantly Supervised Neural Networks](https://mlsanity.com/paper/1804.02445/) (2018)
- [Cross-Domain Document Object Detection: Benchmark Suite and Method](https://mlsanity.com/paper/2003.13197/) (2020)
- [Combining Visual and Textual Features for Semantic Segmentation of Historical Newspapers](https://mlsanity.com/paper/2002.06144/) (2020)
- [Fully Convolutional Neural Networks for Page Segmentation of Historical Document Images](https://mlsanity.com/paper/1711.07695/) (2017)

## Built on by (most cited first, 30 of 40)

- [DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis](https://mlsanity.com/paper/2206.01062/) (2022)
- [Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing](https://mlsanity.com/paper/2506.09965/) (2025)
- [Baichuan-Omni-1.5 Technical Report](https://mlsanity.com/paper/2501.15368/) (2025)
- [Vision Grid Transformer for Document Layout Analysis](https://mlsanity.com/paper/2308.14978/) (2023)
- [M6Doc: A Large-Scale Multi-Format, Multi-Type, Multi-Layout, Multi-Language, Multi-Annotation Category Dataset for Modern Document Layout Analysis](https://mlsanity.com/paper/2305.08719/) (2023)
- [PDFVQA: A New Dataset for Real-World VQA on PDF Documents](https://mlsanity.com/paper/2304.06447/) (2023)
- [LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding](https://mlsanity.com/paper/2605.27365/) (2026)
- [FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture](https://mlsanity.com/paper/2406.11030/) (2024)
- [PP-StructureV2: A Stronger Document Analysis System](https://mlsanity.com/paper/2210.05391/) (2022)
- [Doc-GCN: Heterogeneous Graph Convolutional Networks for Document Layout Analysis](https://mlsanity.com/paper/2208.10970/) (2022)
- [POINTS: Improving Your Vision-language Model with Affordable Strategies](https://mlsanity.com/paper/2409.04828/) (2024)
- [Detect-Order-Construct: A Tree Construction based Approach for Hierarchical Document Structure Analysis](https://mlsanity.com/paper/2401.11874/) (2024)
- [RoDLA: Benchmarking the Robustness of Document Layout Analysis Models](https://mlsanity.com/paper/2403.14442/) (2024)
- [DLAFormer: An End-to-End Transformer For Document Layout Analysis](https://mlsanity.com/paper/2405.11757/) (2024)
- [A Graphical Approach to Document Layout Analysis](https://mlsanity.com/paper/2308.02051/) (2023)
- [Towards End-to-End Semi-Supervised Table Detection with Deformable Transformer](https://mlsanity.com/paper/2305.02769/) (2023)
- [Graph Neural Networks and Representation Embedding for Table Extraction in PDF Documents](https://mlsanity.com/paper/2208.11203/) (2022)
- [ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documents](https://mlsanity.com/paper/2305.14962/) (2023)
- [A Hybrid Approach for Document Layout Analysis in Document images](https://mlsanity.com/paper/2404.17888/) (2024)
- [Towards End-to-End Semi-Supervised Table Detection with Semantic Aligned Matching Transformer](https://mlsanity.com/paper/2405.00187/) (2024)
- [End-to-End Semi-Supervised approach with Modulated Object Queries for Table Detection in Documents](https://mlsanity.com/paper/2405.04971/) (2024)
- [Small Vision-Language Models: A Survey on Compact Architectures and Techniques](https://mlsanity.com/paper/2503.10665/) (2025)
- [DocSAM: Unified Document Image Segmentation via Query Decomposition and Heterogeneous Mixed Learning](https://mlsanity.com/paper/2504.04085/) (2025)
- [GAE: Unleashing Physical Potential of VLM with Generalizable Action Expert](https://mlsanity.com/paper/2510.03896/) (2025)
- [DocAligner: Annotating Real-world Photographic Document Images by Simply Taking Pictures](https://mlsanity.com/paper/2306.05749/) (2023)
- [Reading the unreadable: Creating a dataset of 19th century English newspapers using image-to-text language models](https://mlsanity.com/paper/2502.14901/) (2025)
- [UniHDSA: A Unified Relation Prediction Approach for Hierarchical Document Structure Analysis](https://mlsanity.com/paper/2503.15893/) (2025)
- [Transferability Between Understanding and Generation in Unified Multimodal Models](https://mlsanity.com/paper/2607.04423/) (2026)
- [A Scalable Framework for Table of Contents Extraction from Complex ESG Annual Reports](https://mlsanity.com/paper/2310.18073/) (2023)
- [GraphKD: Exploring Knowledge Distillation Towards Document Object Detection with Structured Graph Creation](https://mlsanity.com/paper/2402.11401/) (2024)

## Abstract (arXiv)

Document layout analysis is crucial for understanding document structures. On this task, vision and semantics of documents, and relations between layout components contribute to the understanding process. Though many works have been proposed to exploit the above information, they show unsatisfactory results. NLP-based methods model layout analysis as a sequence labeling task and show insufficient capabilities in layout modeling. CV-based methods model layout analysis as a detection or segmentation task, but bear limitations of inefficient modality fusion and lack of relation modeling between layout components. To address the above limitations, we propose a unified framework VSR for document layout analysis, combining vision, semantics and relations. VSR supports both NLP-based and CV-based methods. Specifically, we first introduce vision through document image and semantics through text embedding maps. Then, modality-specific visual and semantic features are extracted using a two-stream network, which are adaptively fused to make full use of complementary information. Finally, given component candidates, a relation module based on graph neural network is incorported to model relations between components and output final results. On three popular benchmarks, VSR outperforms previous models by large margins. Code will be released soon.