# Learn to Augment: Joint Data Augmentation and Network Optimization for Text Recognition

arXiv: https://arxiv.org/abs/2003.06606 · HTML: https://mlsanity.com/paper/2003.06606/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2020.

## Built on by (most cited first, 30 of 36)

- [Conditional Text Image Generation with Diffusion Models](https://mlsanity.com/paper/2306.10804/) (2023)
- [PIMNet: A Parallel, Iterative and Mimicking Network for Scene Text Recognition](https://mlsanity.com/paper/2109.04145/) (2021)
- [Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting](https://mlsanity.com/paper/2103.13716/) (2021)
- [CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognition](https://mlsanity.com/paper/2111.11011/) (2021)
- [MetaHTR: Towards Writer-Adaptive Handwritten Text Recognition](https://mlsanity.com/paper/2104.01876/) (2021)
- [Human-Aided Saliency Maps Improve Generalization of Deep Learning](https://mlsanity.com/paper/2105.03492/) (2021)
- [SLOGAN: Handwriting Style Synthesis for Arbitrary-Length and Out-of-Vocabulary Text](https://mlsanity.com/paper/2202.11456/) (2022)
- [Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognition](https://mlsanity.com/paper/2107.12090/) (2021)
- [An End-to-End Robust Point Cloud Semantic Segmentation Network with Single-Step Conditional Diffusion Models](https://mlsanity.com/paper/2411.16308/) (2024)
- [SimAN: Exploring Self-Supervised Representation Learning of Scene Text via Similarity-Aware Normalization](https://mlsanity.com/paper/2203.10492/) (2022)
- [Text is Text, No Matter What: Unifying Text Recognition using Knowledge Distillation](https://mlsanity.com/paper/2107.12087/) (2021)
- [AGTGAN: Unpaired Image Translation for Photographic Ancient Character Generation](https://mlsanity.com/paper/2303.07012/) (2023)
- [Best Practices for a Handwritten Text Recognition System](https://mlsanity.com/paper/2404.11339/) (2024)
- [Learning Instance-Specific Augmentations by Capturing Local Invariances](https://mlsanity.com/paper/2206.00051/) (2022)
- [Seq-UPS: Sequential Uncertainty-aware Pseudo-label Selection for Semi-Supervised Text Recognition](https://mlsanity.com/paper/2209.00641/) (2022)
- [Towards the Unseen: Iterative Text Recognition by Distilling from Errors](https://mlsanity.com/paper/2107.12081/) (2021)
- [RZCR: Zero-shot Character Recognition via Radical-based Reasoning](https://mlsanity.com/paper/2207.05842/) (2022)
- [TextAdaIN: Paying Attention to Shortcut Learning in Text Recognizers](https://mlsanity.com/paper/2105.03906/) (2021)
- [SGBANet: Semantic GAN and Balanced Attention Network for Arbitrarily Oriented Scene Text Recognition](https://mlsanity.com/paper/2207.10256/) (2022)
- [On the Generalization of Handwritten Text Recognition Models](https://mlsanity.com/paper/2411.17332/) (2024)
- [Improving Handwritten OCR with Training Samples Generated by Glyph Conditional Denoising Diffusion Probabilistic Model](https://mlsanity.com/paper/2305.19543/) (2023)
- [Sill-Net: Feature Augmentation with Separated Illumination Representation](https://mlsanity.com/paper/2102.03539/) (2021)
- [Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning](https://mlsanity.com/paper/2108.11912/) (2021)
- [EPiDA: An Easy Plug-in Data Augmentation Framework for High Performance Text Classification](https://mlsanity.com/paper/2204.11205/) (2022)
- [Multi-Granularity Prediction with Learnable Fusion for Scene Text Recognition](https://mlsanity.com/paper/2307.13244/) (2023)
- [Data Augmentation for Scene Text Recognition](https://mlsanity.com/paper/2108.06949/) (2021)
- [Self-supervised Learning for Panoptic Segmentation of Multiple Fruit Flower Species](https://mlsanity.com/paper/2209.04618/) (2022)
- [Self-Supervised Learning for Text Recognition: A Critical Survey](https://mlsanity.com/paper/2407.19889/) (2024)
- [Platypus: A Generalized Specialist Model for Reading Text in Various Forms](https://mlsanity.com/paper/2408.14805/) (2024)
- [MetaWriter: Personalized Handwritten Text Recognition Using Meta-Learned Prompt Tuning](https://mlsanity.com/paper/2505.20513/) (2025)

## Abstract (arXiv)

Handwritten text and scene text suffer from various shapes and distorted patterns. Thus training a robust recognition model requires a large amount of data to cover diversity as much as possible. In contrast to data collection and annotation, data augmentation is a low cost way. In this paper, we propose a new method for text image augmentation. Different from traditional augmentation methods such as rotation, scaling and perspective transformation, our proposed augmentation method is designed to learn proper and efficient data augmentation which is more effective and specific for training a robust recognizer. By using a set of custom fiducial points, the proposed augmentation method is flexible and controllable. Furthermore, we bridge the gap between the isolated processes of data augmentation and network optimization by joint learning. An agent network learns from the output of the recognition network and controls the fiducial points to generate more proper training samples for the recognition network. Extensive experiments on various benchmarks, including regular scene text, irregular scene text and handwritten text, show that the proposed augmentation and the joint learning methods significantly boost the performance of the recognition networks. A general toolkit for geometric augmentation is available.