# DocAligner: Annotating Real-world Photographic Document Images by Simply Taking Pictures

arXiv: https://arxiv.org/abs/2306.05749 · HTML: https://mlsanity.com/paper/2306.05749/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 47 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [DocBank: A Benchmark Dataset for Document Layout Analysis](https://mlsanity.com/paper/2006.01038/) (2020)
- [GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences](https://mlsanity.com/paper/1912.05524/) (2019)
- [Convolutional neural network architecture for geometric matching](https://mlsanity.com/paper/1703.05593/) (2017)
- [TableBank: A Benchmark Dataset for Table Detection and Recognition](https://mlsanity.com/paper/1903.01949/) (2019)
- [Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network](https://mlsanity.com/paper/1706.02337/) (2017)
- [Complicated Table Structure Recognition](https://mlsanity.com/paper/1908.04729/) (2019)
- [DGC-Net: Dense Geometric Correspondence Network](https://mlsanity.com/paper/1810.08393/) (2018)
- [Towards Robust Visual Information Extraction in Real World: New Dataset and Novel Solution](https://mlsanity.com/paper/2102.06732/) (2021)
- [GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Network](https://mlsanity.com/paper/2009.07823/) (2020)
- [CATs++: Boosting Cost Aggregation with Convolutions and Transformers](https://mlsanity.com/paper/2202.06817/) (2022)
- [SCNet: Learning Semantic Correspondence](https://mlsanity.com/paper/1705.04043/) (2017)
- [RANSAC-Flow: generic two-stage image alignment](https://mlsanity.com/paper/2004.01526/) (2020)
- [Parsing Table Structures in the Wild](https://mlsanity.com/paper/2109.02199/) (2021)
- [VSR: A Unified Framework for Document Layout Analysis combining Vision, Semantics and Relations](https://mlsanity.com/paper/2105.06220/) (2021)
- [DocTr: Document Image Transformer for Geometric Unwarping and Illumination Correction](https://mlsanity.com/paper/2110.12942/) (2021)
- [Document Rectification and Illumination Correction using a Patch-based CNN](https://mlsanity.com/paper/1909.09470/) (2019)
- [Recurrent Transformer Networks for Semantic Correspondence](https://mlsanity.com/paper/1810.12155/) (2018)
- [Deep Matching Prior: Test-Time Optimization for Dense Correspondence](https://mlsanity.com/paper/2106.03090/) (2021)
- [Multiview Rectification of Folded Documents](https://mlsanity.com/paper/1606.00166/) (2016)
- [Geometric Representation Learning for Document Image Rectification](https://mlsanity.com/paper/2210.08161/) (2022)
- [PARN: Pyramidal Affine Regression Networks for Dense Semantic Correspondence](https://mlsanity.com/paper/1807.02939/) (2018)

## Built on by (most cited first, 3 of 3)

- [DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming](https://mlsanity.com/paper/2406.19101/) (2024)
- [DocRes: A Generalist Model Toward Unifying Document Image Restoration Tasks](https://mlsanity.com/paper/2405.04408/) (2024)
- [GL-PGENet: A Parameterized Generation Framework for Robust Document Image Enhancement](https://mlsanity.com/paper/2505.22021/) (2025)

## Abstract (arXiv)

Recently, there has been a growing interest in research concerning document image analysis and recognition in photographic scenarios. However, the lack of labeled datasets for this emerging challenge poses a significant obstacle, as manual annotation can be time-consuming and impractical. To tackle this issue, we present DocAligner, a novel method that streamlines the manual annotation process to a simple step of taking pictures. DocAligner achieves this by establishing dense correspondence between photographic document images and their clean counterparts. It enables the automatic transfer of existing annotations in clean document images to photographic ones and helps to automatically acquire labels that are unavailable through manual labeling. Considering the distinctive characteristics of document images, DocAligner incorporates several innovative features. First, we propose a non-rigid pre-alignment technique based on the document's edges, which effectively eliminates interference caused by significant global shifts and repetitive patterns present in document images. Second, to handle large shifts and ensure high accuracy, we introduce a hierarchical aligning approach that combines global and local correlation layers. Furthermore, considering the importance of fine-grained elements in document images, we present a details recurrent refinement module to enhance the output in a high-resolution space. To train DocAligner, we construct a synthetic dataset and introduce a self-supervised learning approach to enhance its robustness for real-world data. Through extensive experiments, we demonstrate the effectiveness of DocAligner and the acquired dataset. Datasets and codes will be publicly available.