# Self-Supervised Representation Learning on Document Images

arXiv: https://arxiv.org/abs/2004.10605 · HTML: https://mlsanity.com/paper/2004.10605/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2020.

## Built on by (most cited first, 5 of 5)

- [SelfDoc: Self-Supervised Document Representation Learning](https://mlsanity.com/paper/2106.03331/) (2021)
- [FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction](https://mlsanity.com/paper/2305.02549/) (2023)
- [VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classification](https://mlsanity.com/paper/2205.12029/) (2022)
- [CICA: Content-Injected Contrastive Alignment for Zero-Shot Document Image Classification](https://mlsanity.com/paper/2405.03660/) (2024)
- [DIVA-DAF: A Deep Learning Framework for Historical Document Image Analysis](https://mlsanity.com/paper/2201.08295/) (2022)

## Abstract (arXiv)

This work analyses the impact of self-supervised pre-training on document images in the context of document image classification. While previous approaches explore the effect of self-supervision on natural images, we show that patch-based pre-training performs poorly on document images because of their different structural properties and poor intra-sample semantic information. We propose two context-aware alternatives to improve performance on the Tobacco-3482 image classification task. We also propose a novel method for self-supervision, which makes use of the inherent multi-modality of documents (image and text), which performs better than other popular self-supervised methods, including supervised ImageNet pre-training, on document image classification scenarios with a limited amount of data.