# Improving Handwritten Text Recognition via 3D Attention and Multi-Scale Training

arXiv: https://arxiv.org/abs/2410.18374 · HTML: https://mlsanity.com/paper/2410.18374/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 24 Oct 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://mlsanity.com/paper/2005.10977/) (2020)
- [Focusing Attention: Towards Accurate Text Recognition in Natural Images](https://mlsanity.com/paper/1709.02054/) (2017)
- [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://mlsanity.com/paper/2108.09661/) (2021)
- [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022)
- [On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention](https://mlsanity.com/paper/1910.04396/) (2019)
- [Chinese Text Recognition with A Pre-Trained CLIP-Like Model Through Image-IDS Aligning](https://mlsanity.com/paper/2309.01083/) (2023)
- [An Efficient End-to-End Neural Model for Handwritten Text Recognition](https://mlsanity.com/paper/1807.07965/) (2018)
- [Recognition of Handwritten Chinese Text by Segmentation: A Segment-annotation-free Approach](https://mlsanity.com/paper/2207.14801/) (2022)
- [Writer-Aware CNN for Parsimonious HMM-Based Offline Handwritten Chinese Text Recognition](https://mlsanity.com/paper/1812.09809/) (2018)
- [Pure Transformer with Integrated Experts for Scene Text Recognition](https://mlsanity.com/paper/2211.04963/) (2022)
- [PageNet: Towards End-to-End Weakly Supervised Page-Level Handwritten Chinese Text Recognition](https://mlsanity.com/paper/2207.14807/) (2022)
- [Recurrent neural network transducer for Japanese and Chinese offline handwritten text recognition](https://mlsanity.com/paper/2106.14459/) (2021)
- [Leveraging Structure Knowledge and Deep Models for the Detection of Abnormal Handwritten Text](https://mlsanity.com/paper/2410.11670/) (2024)

## Abstract (arXiv)

The segmentation-free research efforts for addressing handwritten text recognition can be divided into three categories: connectionist temporal classification (CTC), hidden Markov model and encoder-decoder methods. In this paper, inspired by the above three modeling methods, we propose a new recognition network by using a novel three-dimensional (3D) attention module and global-local context information. Based on the feature maps of the last convolutional layer, a series of 3D blocks with different resolutions are split. Then, these 3D blocks are fed into the 3D attention module to generate sequential visual features. Finally, by fusing the visual features and the corresponding global-local context features, a well-designed representation can be obtained. Main canonical neural units including attention mechanisms, fully-connected layers, recurrent units and convolutional layers are efficiently organized into a network and can be jointly trained by the CTC loss and the cross-entropy loss. Experiments on the latest Chinese handwritten text datasets (the SCUT-HCCDoc and the SCUT-EPT) and one English handwritten text dataset (the IAM) show that the proposed method can achieve comparable results with the state-of-the-art methods. The code is available at https://github.com/Wukong90/3DAttention-MultiScaleTraining-for-HTR.