# Vision Transformer for Fast and Efficient Scene Text Recognition

arXiv: https://arxiv.org/abs/2105.08582 · HTML: https://mlsanity.com/paper/2105.08582/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 18 May 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://mlsanity.com/paper/2003.12294/) (2020)
- [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://mlsanity.com/paper/1811.00751/) (2018)
- [Robust Scene Text Recognition with Automatic Rectification](https://mlsanity.com/paper/1603.03915/) (2016)
- [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://mlsanity.com/paper/2005.10977/) (2020)
- [ESIR: End-to-end Scene Text Recognition via Iterative Image Rectification](https://mlsanity.com/paper/1812.05824/) (2018)
- [Recursive Recurrent Nets with Attention Modeling for OCR in the Wild](https://mlsanity.com/paper/1603.03101/) (2016)
- [NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition](https://mlsanity.com/paper/1806.00926/) (2018)
- [Rosetta: Large scale system for text detection and recognition in images](https://mlsanity.com/paper/1910.05085/) (2019)
- [SCATTER: Selective Context Attentional Scene Text Recognizer](https://mlsanity.com/paper/2003.11288/) (2020)
- [On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention](https://mlsanity.com/paper/1910.04396/) (2019)

## Built on by (most cited first, 30 of 81)

- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://mlsanity.com/paper/2207.06966/) (2022)
- [SVTR: Scene Text Recognition with a Single Visual Model](https://mlsanity.com/paper/2205.00159/) (2022)
- [Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering](https://mlsanity.com/paper/2403.09622/) (2024)
- [DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition](https://mlsanity.com/paper/2203.12273/) (2022)
- [UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models](https://mlsanity.com/paper/2312.04884/) (2023)
- [Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation](https://mlsanity.com/paper/2305.16318/) (2023)
- [TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control](https://mlsanity.com/paper/2410.10133/) (2024)
- [DTrOCR: Decoder-only Transformer for Optical Character Recognition](https://mlsanity.com/paper/2308.15996/) (2023)
- [Multi-Granularity Prediction for Scene Text Recognition](https://mlsanity.com/paper/2209.03592/) (2022)
- [Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Features](https://mlsanity.com/paper/2111.15263/) (2021)
- [CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model](https://mlsanity.com/paper/2305.14014/) (2023)
- [Self-supervised Implicit Glyph Attention for Text Recognition](https://mlsanity.com/paper/2203.03382/) (2022)
- [Levenshtein OCR](https://mlsanity.com/paper/2209.03594/) (2022)
- [On the Cross-dataset Generalization in License Plate Recognition](https://mlsanity.com/paper/2201.00267/) (2022)
- [DreamText: High Fidelity Scene Text Synthesis](https://mlsanity.com/paper/2405.14701/) (2024)
- [Multimodal Semi-Supervised Learning for Text Recognition](https://mlsanity.com/paper/2205.03873/) (2022)
- [CLIPTER: Looking at the Bigger Picture in Scene Text Recognition](https://mlsanity.com/paper/2301.07464/) (2023)
- [Do We Train on Test Data? The Impact of Near-Duplicates on License Plate Recognition](https://mlsanity.com/paper/2304.04653/) (2023)
- [Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition](https://mlsanity.com/paper/2310.04999/) (2023)
- [LISTER: Neighbor Decoding for Length-Insensitive Scene Text Recognition](https://mlsanity.com/paper/2308.12774/) (2023)
- [Supervised Fine-tuning in turn Improves Visual Foundation Models](https://mlsanity.com/paper/2401.10222/) (2024)
- [IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognition](https://mlsanity.com/paper/2312.11923/) (2023)
- [Leveraging Model Fusion for Improved License Plate Recognition](https://mlsanity.com/paper/2309.04331/) (2023)
- [Pure Transformer with Integrated Experts for Scene Text Recognition](https://mlsanity.com/paper/2211.04963/) (2022)
- [A Transformer-based Approach for Arabic Offline Handwritten Text Recognition](https://mlsanity.com/paper/2307.15045/) (2023)
- [DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness](https://mlsanity.com/paper/2412.00151/) (2024)
- [LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer](https://mlsanity.com/paper/2412.13871/) (2024)
- [DiffusionSTR: Diffusion Model for Scene Text Recognition](https://mlsanity.com/paper/2306.16707/) (2023)
- [Multi-Granularity Prediction with Learnable Fusion for Scene Text Recognition](https://mlsanity.com/paper/2307.13244/) (2023)

## Abstract (arXiv)

Scene text recognition (STR) enables computers to read text in natural scenes such as object labels, road signs and instructions. STR helps machines perform informed decisions such as what object to pick, which direction to go, and what is the next step of action. In the body of work on STR, the focus has always been on recognition accuracy. There is little emphasis placed on speed and computational efficiency which are equally important especially for energy-constrained mobile machines. In this paper we propose ViTSTR, an STR with a simple single stage model architecture built on a compute and parameter efficient vision transformer (ViT). On a comparable strong baseline method such as TRBA with accuracy of 84.3%, our small ViTSTR achieves a competitive accuracy of 82.6% (84.2% with data augmentation) at 2.4x speed up, using only 43.4% of the number of parameters and 42.2% FLOPS. The tiny version of ViTSTR achieves 80.3% accuracy (82.1% with data augmentation), at 2.5x the speed, requiring only 10.9% of the number of parameters and 11.9% FLOPS. With data augmentation, our base ViTSTR outperforms TRBA at 85.2% accuracy (83.7% without augmentation) at 2.3x the speed but requires 73.2% more parameters and 61.5% more FLOPS. In terms of trade-offs, nearly all ViTSTR configurations are at or near the frontiers to maximize accuracy, speed and computational efficiency all at the same time.