# Self-Supervised Vision Transformers for Writer Retrieval

arXiv: https://arxiv.org/abs/2409.00751 · HTML: https://mlsanity.com/paper/2409.00751/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 1 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Spreading vectors for similarity search](https://mlsanity.com/paper/1806.03198/) (2018)
- [What to Hide from Your Students: Attention-Guided Masked Image Modeling](https://mlsanity.com/paper/2203.12719/) (2022)
- [Unsupervised Feature Learning for Writer Identification and Writer Retrieval](https://mlsanity.com/paper/1705.09369/) (2017)
- [Text-DIAE: A Self-Supervised Degradation Invariant Autoencoders for Text Recognition and Document Enhancement](https://mlsanity.com/paper/2203.04814/) (2022)
- [Encoding CNN Activations for Writer Recognition](https://mlsanity.com/paper/1712.07923/) (2017)
- [Generalized Max Pooling](https://mlsanity.com/paper/1406.0312/) (2014)
- [ICDAR 2019 Competition on Image Retrieval for Historical Handwritten Documents](https://mlsanity.com/paper/1912.03713/) (2019)
- [Writer Identification and Writer Retrieval Based on NetVLAD with Re-ranking](https://mlsanity.com/paper/2012.06186/) (2020)
- [Towards Writer Retrieval for Historical Datasets](https://mlsanity.com/paper/2305.05358/) (2023)
- [Re-ranking for Writer Identification and Writer Retrieval](https://mlsanity.com/paper/2007.07101/) (2020)

## Built on by (most cited first, 4 of 4)

- [Towards the Influence of Text Quantity on Writer Retrieval](https://mlsanity.com/paper/2506.07566/) (2025)
- [BullingerDB: A Dataset for Handwritten Text Recognition and Writer Retrieval](https://mlsanity.com/paper/2605.30235/) (2026)
- [Handwriting Extraction and Analysis of Signature Lists in Swiss Popular Initiatives](https://mlsanity.com/paper/2606.05018/) (2026)
- [TextileNet: Towards Zero-shot Text-style Segmentation of Manuscripts](https://mlsanity.com/paper/2607.09299/) (2026)

## Abstract (arXiv)

While methods based on Vision Transformers (ViT) have achieved state-of-the-art performance in many domains, they have not yet been applied successfully in the domain of writer retrieval. The field is dominated by methods using handcrafted features or features extracted from Convolutional Neural Networks. In this work, we bridge this gap and present a novel method that extracts features from a ViT and aggregates them using VLAD encoding. The model is trained in a self-supervised fashion without any need for labels. We show that extracting local foreground features is superior to using the ViT's class token in the context of writer retrieval. We evaluate our method on two historical document collections. We set a new state-at-of-art performance on the Historical-WI dataset (83.1% mAP), and the HisIR19 dataset (95.0% mAP). Additionally, we demonstrate that our ViT feature extractor can be directly applied to modern datasets such as the CVL database (98.6% mAP) without any fine-tuning.