# ESIR: End-to-end Scene Text Recognition via Iterative Image Rectification

arXiv: https://arxiv.org/abs/1812.05824 · HTML: https://mlsanity.com/paper/1812.05824/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2018.

## Built on by (most cited first, 30 of 79)

- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://mlsanity.com/paper/2108.09661/) (2021)
- [ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting](https://mlsanity.com/paper/2105.03620/) (2021)
- [Vision Transformer for Fast and Efficient Scene Text Recognition](https://mlsanity.com/paper/2105.08582/) (2021)
- [Diverse Image Inpainting with Bidirectional and Autoregressive Transformers](https://mlsanity.com/paper/2104.12335/) (2021)
- [PAN++: Towards Efficient and Accurate End-to-End Spotting of Arbitrarily-Shaped Text](https://mlsanity.com/paper/2105.00405/) (2021)
- [What If We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labels](https://mlsanity.com/paper/2103.04400/) (2021)
- [Conditional Text Image Generation with Diffusion Models](https://mlsanity.com/paper/2306.10804/) (2023)
- [Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Study](https://mlsanity.com/paper/2112.15093/) (2021)
- [PGNet: Real-time Arbitrarily-Shaped Text Spotting with Point Gathering Network](https://mlsanity.com/paper/2104.05458/) (2021)
- [Rethinking Text Line Recognition Models](https://mlsanity.com/paper/2104.07787/) (2021)
- [Unbalanced Feature Transport for Exemplar-based Image Translation](https://mlsanity.com/paper/2106.10482/) (2021)
- [PIMNet: A Parallel, Iterative and Mimicking Network for Scene Text Recognition](https://mlsanity.com/paper/2109.04145/) (2021)
- [ABINet++: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Spotting](https://mlsanity.com/paper/2211.10578/) (2022)
- [CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognition](https://mlsanity.com/paper/2111.11011/) (2021)
- [Multi-Granularity Prediction for Scene Text Recognition](https://mlsanity.com/paper/2209.03592/) (2022)
- [Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Features](https://mlsanity.com/paper/2111.15263/) (2021)
- [Blind Image Super-Resolution via Contrastive Representation Learning](https://mlsanity.com/paper/2107.00708/) (2021)
- [GMLight: Lighting Estimation via Geometric Distribution Approximation](https://mlsanity.com/paper/2102.10244/) (2021)
- [Primitive Representation Learning for Scene Text Recognition](https://mlsanity.com/paper/2105.04286/) (2021)
- [TPSNet: Reverse Thinking of Thin Plate Splines for Arbitrary Shape Scene Text Representation](https://mlsanity.com/paper/2110.12826/) (2021)
- [Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognition](https://mlsanity.com/paper/2107.12090/) (2021)
- [Levenshtein OCR](https://mlsanity.com/paper/2209.03594/) (2022)
- [Bi-level Feature Alignment for Versatile Image Translation and Manipulation](https://mlsanity.com/paper/2107.03021/) (2021)
- [Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognition](https://mlsanity.com/paper/2303.04291/) (2023)
- [Sparse Needlets for Lighting Estimation with Spherical Transport Loss](https://mlsanity.com/paper/2106.13090/) (2021)
- [Multimodal Semi-Supervised Learning for Text Recognition](https://mlsanity.com/paper/2205.03873/) (2022)
- [PERT: A Progressively Region-based Network for Scene Text Removal](https://mlsanity.com/paper/2106.13029/) (2021)
- [Scalable Mask Annotation for Video Text Spotting](https://mlsanity.com/paper/2305.01443/) (2023)

## Abstract (arXiv)

Automated recognition of texts in scenes has been a research challenge for years, largely due to the arbitrary variation of text appearances in perspective distortion, text line curvature, text styles and different types of imaging artifacts. The recent deep networks are capable of learning robust representations with respect to imaging artifacts and text style changes, but still face various problems while dealing with scene texts with perspective and curvature distortions. This paper presents an end-to-end trainable scene text recognition system (ESIR) that iteratively removes perspective distortion and text line curvature as driven by better scene text recognition performance. An innovative rectification network is developed which employs a novel line-fitting transformation to estimate the pose of text lines in scenes. In addition, an iterative rectification pipeline is developed where scene text distortions are corrected iteratively towards a fronto-parallel view. The ESIR is also robust to parameter initialization and the training needs only scene text images and word-level annotations as required by most scene text recognition systems. Extensive experiments over a number of public datasets show that the proposed ESIR is capable of rectifying scene text distortions accurately, achieving superior recognition performance for both normal scene text images and those suffering from perspective and curvature distortions.