# Evaluating Sequence-to-Sequence Models for Handwritten Text Recognition

arXiv: https://arxiv.org/abs/1903.07377 · HTML: https://mlsanity.com/paper/1903.07377/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2019.

## Built on by (most cited first, 30 of 58)

- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022)
- [Rethinking Text Line Recognition Models](https://mlsanity.com/paper/2104.07787/) (2021)
- [DTrOCR: Decoder-only Transformer for Optical Character Recognition](https://mlsanity.com/paper/2308.15996/) (2023)
- [HTR-VT: Handwritten Text Recognition with Vision Transformer](https://mlsanity.com/paper/2409.08573/) (2024)
- [MathWriting: A Dataset For Handwritten Mathematical Expression Recognition](https://mlsanity.com/paper/2404.10690/) (2024)
- [SPAN: a Simple Predict & Align Network for Handwritten Paragraph Recognition](https://mlsanity.com/paper/2102.08742/) (2021)
- [Rescoring Sequence-to-Sequence Models for Text Line Recognition with CTC-Prefixes](https://mlsanity.com/paper/2110.05909/) (2021)
- [SmartPatch: Improving Handwritten Word Imitation with Patch Discriminators](https://mlsanity.com/paper/2105.10528/) (2021)
- [Transformer-Based Approach for Joint Handwriting and Named Entity Recognition in Historical documents](https://mlsanity.com/paper/2112.04189/) (2021)
- [The LAM Dataset: A Novel Benchmark for Line-Level Handwritten Text Recognition](https://mlsanity.com/paper/2208.07682/) (2022)
- [Best Practices for a Handwritten Text Recognition System](https://mlsanity.com/paper/2404.11339/) (2024)
- [Boosting Modern and Historical Handwritten Text Recognition with Deformable Convolutions](https://mlsanity.com/paper/2208.08109/) (2022)
- [Benchmarking Online Sequence-to-Sequence and Character-based Handwriting Recognition from IMU-Enhanced Pens](https://mlsanity.com/paper/2202.07036/) (2022)
- [AttentionHTR: Handwritten Text Recognition Based on Attention Encoder-Decoder Networks](https://mlsanity.com/paper/2201.09390/) (2022)
- [Handwritten Text Recognition: A Survey](https://mlsanity.com/paper/2502.08417/) (2025)
- [Star Temporal Classification: Sequence Classification with Partially Labeled Data](https://mlsanity.com/paper/2201.12208/) (2022)
- [On the Generalization of Handwritten Text Recognition Models](https://mlsanity.com/paper/2411.17332/) (2024)
- [A Transformer-based Approach for Arabic Offline Handwritten Text Recognition](https://mlsanity.com/paper/2307.15045/) (2023)
- [DANIEL: A fast Document Attention Network for Information Extraction and Labelling of handwritten documents](https://mlsanity.com/paper/2407.09103/) (2024)
- [HATFormer: Historic Handwritten Arabic Text Recognition with Transformers](https://mlsanity.com/paper/2410.02179/) (2024)
- [Improving Handwritten OCR with Training Samples Generated by Glyph Conditional Denoising Diffusion Probabilistic Model](https://mlsanity.com/paper/2305.19543/) (2023)
- [TS-Net: OCR Trained to Switch Between Text Transcription Styles](https://mlsanity.com/paper/2103.05489/) (2021)
- [Learning Non-Stationary Time-Series with Dynamic Pattern Extractions](https://mlsanity.com/paper/2111.10559/) (2021)
- [Representing Online Handwriting for Recognition in Large Vision-Language Models](https://mlsanity.com/paper/2402.15307/) (2024)
- [Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting Recognition](https://mlsanity.com/paper/2407.13559/) (2024)
- [Auxiliary Cross-Modal Representation Learning with Triplet Loss Functions for Online Handwriting Recognition](https://mlsanity.com/paper/2202.07901/) (2022)
- [Easter2.0: Improving convolutional models for handwritten text recognition](https://mlsanity.com/paper/2205.14879/) (2022)
- [Towards Writing Style Adaptation in Handwriting Recognition](https://mlsanity.com/paper/2302.06318/) (2023)
- [Transformers and Language Models in Form Understanding: A Comprehensive Review of Scanned Document Analysis](https://mlsanity.com/paper/2403.04080/) (2024)

## Abstract (arXiv)

Encoder-decoder models have become an effective approach for sequence learning tasks like machine translation, image captioning and speech recognition, but have yet to show competitive results for handwritten text recognition. To this end, we propose an attention-based sequence-to-sequence model. It combines a convolutional neural network as a generic feature extractor with a recurrent neural network to encode both the visual information, as well as the temporal context between characters in the input image, and uses a separate recurrent neural network to decode the actual character sequence. We make experimental comparisons between various attention mechanisms and positional encodings, in order to find an appropriate alignment between the input and output sequence. The model can be trained end-to-end and the optional integration of a hybrid loss allows the encoder to retain an interpretable and usable output, if desired. We achieve competitive results on the IAM and ICFHR2016 READ data sets compared to the state-of-the-art without the use of a language model, and we significantly improve over any recent sequence-to-sequence approaches.