# HTR-VT: Handwritten Text Recognition with Vision Transformer

arXiv: https://arxiv.org/abs/2409.08573 · HTML: https://mlsanity.com/paper/2409.08573/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 57 references that are in the map.
- First posted on arXiv 13 Sep 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [How Much Position Information Do Convolutional Neural Networks Encode?](https://mlsanity.com/paper/2001.08248/) (2020)
- [Efficient Training of Visual Transformers with Small Datasets](https://mlsanity.com/paper/2106.03746/) (2021)
- [Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network](https://mlsanity.com/paper/1706.02337/) (2017)
- [Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition](https://mlsanity.com/paper/2005.13044/) (2020)
- [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022)
- [Evaluating Sequence-to-Sequence Models for Handwritten Text Recognition](https://mlsanity.com/paper/1903.07377/) (2019)
- [OrigamiNet: Weakly-Supervised, Segmentation-Free, One-Step, Full Page Text Recognition by learning to unfold](https://mlsanity.com/paper/2006.07491/) (2020)
- [End-to-end Handwritten Paragraph Text Recognition Using a Vertical Attention Network](https://mlsanity.com/paper/2012.03868/) (2020)
- [Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition](https://mlsanity.com/paper/1604.08352/) (2016)
- [Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention](https://mlsanity.com/paper/1604.03286/) (2016)
- [Handwriting Recognition in Low-resource Scripts using Adversarial Learning](https://mlsanity.com/paper/1811.01396/) (2018)
- [Accurate, Data-Efficient, Unconstrained Text Recognition with Convolutional Neural Networks](https://mlsanity.com/paper/1812.11894/) (2018)

## Built on by (most cited first, 24 of 24)

- [Handwritten Text Recognition: A Survey](https://mlsanity.com/paper/2502.08417/) (2025)
- [Benchmarking Large Language Models for Handwritten Text Recognition](https://mlsanity.com/paper/2503.15195/) (2025)
- [On the Generalization of Handwritten Text Recognition Models](https://mlsanity.com/paper/2411.17332/) (2024)
- [A Human-in-the-Loop Label Error Detection Framework Applied to Arabic-Script HTR Datasets](https://mlsanity.com/paper/2601.16713/) (2026)
- [Cross-Lingual Learning within Arabic Script for Low-Resource HTR](https://mlsanity.com/paper/2605.02089/) (2026)
- [HTR-ConvText: Leveraging Convolution and Textual Information for Handwritten Text Recognition](https://mlsanity.com/paper/2512.05021/) (2025)
- [Enhancing IMU-Based Online Handwriting Recognition via Contrastive Learning with Zero Inference Overhead](https://mlsanity.com/paper/2602.07049/) (2026)
- [DRetHTR: Linear-Time Decoder-Only Retentive Network for Handwritten Text Recognition](https://mlsanity.com/paper/2602.17387/) (2026)
- [A Text Recognition Dataset from Sahidic Coptic Ancient Manuscripts](https://mlsanity.com/paper/2606.15987/) (2026)
- [Preserving Privacy Without Compromising Accuracy: Machine Unlearning for Handwritten Text Recognition](https://mlsanity.com/paper/2504.08616/) (2025)
- [The End Of Universal Lifelong Identifiers: Identity Systems For The AI Era](https://mlsanity.com/paper/2506.02027/) (2025)
- [A Transformer Based Handwriting Recognition System Jointly Using Online and Offline Features](https://mlsanity.com/paper/2506.20255/) (2025)
- [Improving OCR for Historical Texts of Multiple Languages](https://mlsanity.com/paper/2508.10356/) (2025)
- [FP-THD: Full page transcription of historical documents](https://mlsanity.com/paper/2601.17040/) (2026)
- [PoM: A Linear-Time Replacement for Attention with the Polynomial Mixer](https://mlsanity.com/paper/2604.06129/) (2026)
- [Zero-Shot Synthetic-to-Real Handwritten Text Recognition via Task Analogies](https://mlsanity.com/paper/2604.09713/) (2026)
- [Multi-Head Attention based interaction-aware architecture for Bangla Handwritten Character Recognition: Introducing a Primary Dataset](https://mlsanity.com/paper/2604.09717/) (2026)
- [A Dataset for the Recognition of Historical and Handwritten Music Scores in Western Notation](https://mlsanity.com/paper/2605.18436/) (2026)
- [Leveraging Morphology for Historical Script Metrological Analysis](https://mlsanity.com/paper/2606.09446/) (2026)
- [Performance Gap Analysis between Latin and Arabic Scripts HTR](https://mlsanity.com/paper/2606.18884/) (2026)
- [UniLipi: A Unified Multi-Script OCR for Historical Indic Manuscripts](https://mlsanity.com/paper/2608.28195/) (2026)
- [Automated pipeline for herbarium label digitization](https://mlsanity.com/paper/2608.28676/) (2026)
- [ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts](https://mlsanity.com/paper/2609.12705/) (2026)
- [A Free Lunch? Adapting PP-OCRv6 for Historical Text Recognition](https://mlsanity.com/paper/2609.20064/) (2026)

## Abstract (arXiv)

We explore the application of Vision Transformer (ViT) for handwritten text recognition. The limited availability of labeled data in this domain poses challenges for achieving high performance solely relying on ViT. Previous transformer-based models required external data or extensive pre-training on large datasets to excel. To address this limitation, we introduce a data-efficient ViT method that uses only the encoder of the standard transformer. We find that incorporating a Convolutional Neural Network (CNN) for feature extraction instead of the original patch embedding and employ Sharpness-Aware Minimization (SAM) optimizer to ensure that the model can converge towards flatter minima and yield notable enhancements. Furthermore, our introduction of the span mask technique, which masks interconnected features in the feature map, acts as an effective regularizer. Empirically, our approach competes favorably with traditional CNN-based models on small datasets like IAM and READ2016. Additionally, it establishes a new benchmark on the LAM dataset, currently the largest dataset with 19,830 training text lines. The code is publicly available at: https://github.com/YutingLi0606/HTR-VT.