# Fine-tuning Handwriting Recognition systems with Temporal Dropout

arXiv: https://arxiv.org/abs/2102.00511 · HTML: https://mlsanity.com/paper/2102.00511/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 31 Jan 2021.

## Builds on (most cited first, 15 of 15)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Efficient Object Localization Using Convolutional Networks](https://mlsanity.com/paper/1411.4280/) (2014)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://mlsanity.com/paper/1512.05287/) (2015)
- [Adding Gradient Noise Improves Learning for Very Deep Networks](https://mlsanity.com/paper/1511.06807/) (2015)
- [Shake-Shake regularization](https://mlsanity.com/paper/1705.07485/) (2017)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [Dropout improves Recurrent Neural Networks for Handwriting Recognition](https://mlsanity.com/paper/1312.4569/) (2013)
- [Recurrent Dropout without Memory Loss](https://mlsanity.com/paper/1603.05118/) (2016)
- [Handwriting Recognition of Historical Documents with few labeled data](https://mlsanity.com/paper/1811.07768/) (2018)
- [Whiteout: Gaussian Adaptive Noise Regularization in Deep Neural Networks](https://mlsanity.com/paper/1612.01490/) (2016)
- [Shakeout: A New Approach to Regularized Deep Neural Network Training](https://mlsanity.com/paper/1904.06593/) (2019)

## Built on by (most cited first, 2 of 2)

- [UTRNet: High-Resolution Urdu Text Recognition In Printed Documents](https://mlsanity.com/paper/2306.15782/) (2023)
- [FP-THD: Full page transcription of historical documents](https://mlsanity.com/paper/2601.17040/) (2026)

## Abstract (arXiv)

This paper introduces a novel method to fine-tune handwriting recognition systems based on Recurrent Neural Networks (RNN). Long Short-Term Memory (LSTM) networks are good at modeling long sequences but they tend to overfit over time. To improve the system's ability to model sequences, we propose to drop information at random positions in the sequence. We call our approach Temporal Dropout (TD). We apply TD at the image level as well to internal network representation. We show that TD improves the results on two different datasets. Our method outperforms previous state-of-the-art on Rodrigo dataset.