# SwinDocSegmenter: An End-to-End Unified Domain Adaptive Transformer for Document Instance Segmentation

arXiv: https://arxiv.org/abs/2305.04609 · HTML: https://mlsanity.com/paper/2305.04609/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 8 May 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection](https://mlsanity.com/paper/1904.09730/) (2019)
- [DiT: Self-supervised Pre-training for Document Image Transformer](https://mlsanity.com/paper/2203.02378/) (2022)
- [DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis](https://mlsanity.com/paper/2206.01062/) (2022)
- [Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer](https://mlsanity.com/paper/2102.09550/) (2021)
- [TableBank: A Benchmark Dataset for Table Detection and Recognition](https://mlsanity.com/paper/1903.01949/) (2019)
- [CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents](https://mlsanity.com/paper/2004.12629/) (2020)
- [StrucTexT: Structured Text Understanding with Multi-Modal Transformers](https://mlsanity.com/paper/2108.02923/) (2021)
- [Unified Pretraining Framework for Document Understanding](https://mlsanity.com/paper/2204.10939/) (2022)
- [XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document Understanding](https://mlsanity.com/paper/2203.06947/) (2022)
- [LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis](https://mlsanity.com/paper/2103.15348/) (2021)
- [dhSegment: A generic deep-learning approach for document segmentation](https://mlsanity.com/paper/1804.10371/) (2018)
- [A Large Dataset of Historical Japanese Documents with Complex Layouts](https://mlsanity.com/paper/2004.08686/) (2020)
- [Cross-Domain Document Object Detection: Benchmark Suite and Method](https://mlsanity.com/paper/2003.13197/) (2020)
- [Graphical Object Detection in Document Images](https://mlsanity.com/paper/2008.10843/) (2020)
- [DocSegTr: An Instance-Level End-to-End Document Image Segmentation Transformer](https://mlsanity.com/paper/2201.11438/) (2022)
- [Convolutional Neural Networks for Page Segmentation of Historical Document Images](https://mlsanity.com/paper/1704.01474/) (2017)

## Built on by (most cited first, 13 of 13)

- [OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations](https://mlsanity.com/paper/2412.07626/) (2024)
- [RoDLA: Benchmarking the Robustness of Document Layout Analysis Models](https://mlsanity.com/paper/2403.14442/) (2024)
- ['Eclair -- Extracting Content and Layout with Integrated Reading Order for Documents](https://mlsanity.com/paper/2502.04223/) (2025)
- [GeoContrastNet: Contrastive Key-Value Edge Learning for Language-Agnostic Document Understanding](https://mlsanity.com/paper/2405.03104/) (2024)
- [UniHDSA: A Unified Relation Prediction Approach for Hierarchical Document Structure Analysis](https://mlsanity.com/paper/2503.15893/) (2025)
- [GraphKD: Exploring Knowledge Distillation Towards Document Object Detection with Structured Graph Creation](https://mlsanity.com/paper/2402.11401/) (2024)
- [UnSupDLA: Towards Unsupervised Document Layout Analysis](https://mlsanity.com/paper/2406.06236/) (2024)
- [Class-Agnostic Region-of-Interest Matching in Document Images](https://mlsanity.com/paper/2506.21055/) (2025)
- [Harnessing the Power of Multi-Lingual Datasets for Pre-training: Towards Enhancing Text Spotting Performance](https://mlsanity.com/paper/2310.00917/) (2023)
- [DistilDoc: Knowledge Distillation for Visually-Rich Document Applications](https://mlsanity.com/paper/2406.08226/) (2024)
- [LayeredDoc: Domain Adaptive Document Restoration with a Layer Separation Approach](https://mlsanity.com/paper/2406.08610/) (2024)
- [IndicDLP: A Foundational Dataset for Multi-Lingual and Multi-Domain Document Layout Parsing](https://mlsanity.com/paper/2512.20236/) (2025)
- [PromptDLA: A Domain-aware Prompt Document Layout Analysis Framework with Descriptive Knowledge as a Cue](https://mlsanity.com/paper/2603.09414/) (2026)

## Abstract (arXiv)

Instance-level segmentation of documents consists in assigning a class-aware and instance-aware label to each pixel of the image. It is a key step in document parsing for their understanding. In this paper, we present a unified transformer encoder-decoder architecture for en-to-end instance segmentation of complex layouts in document images. The method adapts a contrastive training with a mixed query selection for anchor initialization in the decoder. Later on, it performs a dot product between the obtained query embeddings and the pixel embedding map (coming from the encoder) for semantic reasoning. Extensive experimentation on competitive benchmarks like PubLayNet, PRIMA, Historical Japanese (HJ), and TableBank demonstrate that our model with SwinL backbone achieves better segmentation performance than the existing state-of-the-art approaches with the average precision of 93.72, 54.39, 84.65 and 98.04 respectively under one billion parameters. The code is made publicly available at: hrefhttps://github.com/ayanban011/SwinDocSegmentergithub.com/ayanban011/SwinDocSegmenter