# Vision Grid Transformer for Document Layout Analysis

arXiv: https://arxiv.org/abs/2308.14978 · HTML: https://mlsanity.com/paper/2308.14978/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 53 references that are in the map.
- First posted on arXiv 29 Aug 2023.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://mlsanity.com/paper/1912.13318/) (2019)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding](https://mlsanity.com/paper/2012.14740/) (2020)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [DocBank: A Benchmark Dataset for Document Layout Analysis](https://mlsanity.com/paper/2006.01038/) (2020)
- [DiT: Self-supervised Pre-training for Document Image Transformer](https://mlsanity.com/paper/2203.02378/) (2022)
- [Scene Text Detection and Recognition: The Deep Learning Era](https://mlsanity.com/paper/1811.04256/) (2018)
- [Chargrid: Towards Understanding 2D Documents](https://mlsanity.com/paper/1809.08799/) (2018)
- [CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents](https://mlsanity.com/paper/2004.12629/) (2020)
- [Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network](https://mlsanity.com/paper/1706.02337/) (2017)
- [StrucTexT: Structured Text Understanding with Multi-Modal Transformers](https://mlsanity.com/paper/2108.02923/) (2021)
- [Unified Pretraining Framework for Document Understanding](https://mlsanity.com/paper/2204.10939/) (2022)
- [BERTgrid: Contextualized Embedding for 2D Document Representation and Understanding](https://mlsanity.com/paper/1909.04948/) (2019)
- [Document AI: Benchmarks, Models and Applications](https://mlsanity.com/paper/2111.08609/) (2021)

## Built on by (most cited first, 30 of 47)

- [MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing](https://mlsanity.com/paper/2509.22186/) (2025)
- [LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding](https://mlsanity.com/paper/2404.05225/) (2024)
- [MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm](https://mlsanity.com/paper/2506.05218/) (2025)
- [DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception](https://mlsanity.com/paper/2410.12628/) (2024)
- [VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning](https://mlsanity.com/paper/2505.12081/) (2025)
- [dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model](https://mlsanity.com/paper/2512.02498/) (2025)
- [Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction](https://mlsanity.com/paper/2410.21169/) (2024)
- [DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models](https://mlsanity.com/paper/2406.11633/) (2024)
- [PP-DocLayout: A Unified Document Layout Detection Model to Accelerate Large-Scale Data Construction](https://mlsanity.com/paper/2503.17213/) (2025)
- [OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models](https://mlsanity.com/paper/2502.16161/) (2025)
- [A Simple yet Effective Layout Token in Large Language Models for Document Understanding](https://mlsanity.com/paper/2503.18434/) (2025)
- [Benchmarking Graph Neural Networks for Document Layout Analysis in Public Affairs](https://mlsanity.com/paper/2505.14699/) (2025)
- [MultiDocFusion: Hierarchical and Multimodal Chunking Pipeline for Enhanced RAG on Long Industrial Documents](https://mlsanity.com/paper/2604.12352/) (2026)
- [TextBlockV2: Towards Precise-Detection-Free Scene Text Spotting with Pre-trained Language Model](https://mlsanity.com/paper/2403.10047/) (2024)
- [ProcTag: Process Tagging for Assessing the Efficacy of Document Instruction Data](https://mlsanity.com/paper/2407.12358/) (2024)
- [MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios](https://mlsanity.com/paper/2603.28130/) (2026)
- [DocSAM: Unified Document Image Segmentation via Query Decomposition and Heterogeneous Mixed Learning](https://mlsanity.com/paper/2504.04085/) (2025)
- [Infinity-Parser2 Technical Report](https://mlsanity.com/paper/2607.07836/) (2026)
- [MonkeyOCRv2: A Visual-Text Foundation Model for Document AI](https://mlsanity.com/paper/2607.11562/) (2026)
- [CREPE: Coordinate-Aware End-to-End Document Parser](https://mlsanity.com/paper/2405.00260/) (2024)
- [Reading the unreadable: Creating a dataset of 19th century English newspapers using image-to-text language models](https://mlsanity.com/paper/2502.14901/) (2025)
- [GraphKD: Exploring Knowledge Distillation Towards Document Object Detection with Structured Graph Creation](https://mlsanity.com/paper/2402.11401/) (2024)
- [UnSupDLA: Towards Unsupervised Document Layout Analysis](https://mlsanity.com/paper/2406.06236/) (2024)
- [SFDLA: Source-Free Document Layout Analysis](https://mlsanity.com/paper/2503.18742/) (2025)
- [Class-Agnostic Region-of-Interest Matching in Document Images](https://mlsanity.com/paper/2506.21055/) (2025)
- [DREAM: Document Reconstruction via End-to-end Autoregressive Model](https://mlsanity.com/paper/2507.05805/) (2025)
- [OmniDocLayout: Towards Diverse Document Layout Generation via Coarse-to-Fine LLM Learning](https://mlsanity.com/paper/2510.26213/) (2025)
- [Object Recognition from Scientific Document based on Compartment Refinement Framework](https://mlsanity.com/paper/2312.09038/) (2023)
- [DistilDoc: Knowledge Distillation for Visually-Rich Document Applications](https://mlsanity.com/paper/2406.08226/) (2024)
- [Exploring OCR-augmented Generation for Bilingual VQA](https://mlsanity.com/paper/2510.02543/) (2025)

## Abstract (arXiv)

Document pre-trained models and grid-based models have proven to be very effective on various tasks in Document AI. However, for the document layout analysis (DLA) task, existing document pre-trained models, even those pre-trained in a multi-modal fashion, usually rely on either textual features or visual features. Grid-based models for DLA are multi-modality but largely neglect the effect of pre-training. To fully leverage multi-modal information and exploit pre-training techniques to learn better representation for DLA, in this paper, we present VGT, a two-stream Vision Grid Transformer, in which Grid Transformer (GiT) is proposed and pre-trained for 2D token-level and segment-level semantic understanding. Furthermore, a new dataset named D4LA, which is so far the most diverse and detailed manually-annotated benchmark for document layout analysis, is curated and released. Experiment results have illustrated that the proposed VGT model achieves new state-of-the-art results on DLA tasks, e.g. PubLayNet (95.7%rightarrow96.2%), DocBank (79.6%rightarrow84.1%), and D4LA (67.7%rightarrow68.8%). The code and models as well as the D4LA dataset will be made publicly available ~urlhttps://github.com/AlibabaResearch/AdvancedLiterateMachinery.