# Multi-Granularity Prediction for Scene Text Recognition

arXiv: https://arxiv.org/abs/2209.03592 · HTML: https://mlsanity.com/paper/2209.03592/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 59 references that are in the map.
- First posted on arXiv 8 Sep 2022.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes](https://mlsanity.com/paper/1908.08207/) (2019)
- [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://mlsanity.com/paper/2003.12294/) (2020)
- [TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?](https://mlsanity.com/paper/2106.11297/) (2021)
- [Reading Text in the Wild with Convolutional Neural Networks](https://mlsanity.com/paper/1412.1842/) (2014)
- [Robust Scene Text Recognition with Automatic Rectification](https://mlsanity.com/paper/1603.03915/) (2016)
- [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://mlsanity.com/paper/2005.10977/) (2020)
- [Scene Text Detection and Recognition: The Deep Learning Era](https://mlsanity.com/paper/1811.04256/) (2018)
- [Focusing Attention: Towards Accurate Text Recognition in Natural Images](https://mlsanity.com/paper/1709.02054/) (2017)
- [Decoupled Attention Network for Text Recognition](https://mlsanity.com/paper/1912.10205/) (2019)
- [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://mlsanity.com/paper/2108.09661/) (2021)
- [RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition](https://mlsanity.com/paper/2007.07542/) (2020)
- [Vision Transformer for Fast and Efficient Scene Text Recognition](https://mlsanity.com/paper/2105.08582/) (2021)
- [ESIR: End-to-end Scene Text Recognition via Iterative Image Rectification](https://mlsanity.com/paper/1812.05824/) (2018)
- [Recursive Recurrent Nets with Attention Modeling for OCR in the Wild](https://mlsanity.com/paper/1603.03101/) (2016)

## Built on by (most cited first, 30 of 36)

- [MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents](https://mlsanity.com/paper/2406.08184/) (2024)
- [Conditional Text Image Generation with Diffusion Models](https://mlsanity.com/paper/2306.10804/) (2023)
- [Vision Grid Transformer for Document Layout Analysis](https://mlsanity.com/paper/2308.14978/) (2023)
- [GeoLayoutLM: Geometric Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2304.10759/) (2023)
- [Self-supervised Character-to-Character Distillation for Text Recognition](https://mlsanity.com/paper/2211.00288/) (2022)
- [SVTRv2: CTC Beats Encoder-Decoder Models in Scene Text Recognition](https://mlsanity.com/paper/2411.15858/) (2024)
- [Context Perception Parallel Decoder for Scene Text Recognition](https://mlsanity.com/paper/2307.12270/) (2023)
- [Linguistic More: Taking a Further Step toward Efficient and Accurate Scene Text Recognition](https://mlsanity.com/paper/2305.05140/) (2023)
- [Instruction-Guided Scene Text Recognition](https://mlsanity.com/paper/2401.17851/) (2024)
- [Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition](https://mlsanity.com/paper/2310.04999/) (2023)
- [Layout Agnostic Scene Text Image Synthesis with Diffusion Models](https://mlsanity.com/paper/2406.01062/) (2024)
- [LISTER: Neighbor Decoding for Length-Insensitive Scene Text Recognition](https://mlsanity.com/paper/2308.12774/) (2023)
- [UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters](https://mlsanity.com/paper/2512.21095/) (2025)
- [IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognition](https://mlsanity.com/paper/2312.11923/) (2023)
- [Class-Aware Mask-Guided Feature Refinement for Scene Text Recognition](https://mlsanity.com/paper/2402.13643/) (2024)
- [Towards Robust Scene Text Image Super-resolution via Explicit Location Enhancement](https://mlsanity.com/paper/2307.09749/) (2023)
- [Hierarchical Text Spotter for Joint Text Spotting and Layout Analysis](https://mlsanity.com/paper/2310.17674/) (2023)
- [Self-Supervised Pre-training with Symmetric Superimposition Modeling for Scene Text Recognition](https://mlsanity.com/paper/2405.05841/) (2024)
- [Multi-Granularity Prediction with Learnable Fusion for Scene Text Recognition](https://mlsanity.com/paper/2307.13244/) (2023)
- [Masked and Permuted Implicit Context Learning for Scene Text Recognition](https://mlsanity.com/paper/2305.16172/) (2023)
- [Cross-Lingual Learning in Multilingual Scene Text Recognition](https://mlsanity.com/paper/2312.10806/) (2023)
- [ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning](https://mlsanity.com/paper/2507.02200/) (2025)
- [Sequential Visual and Semantic Consistency for Semi-supervised Text Recognition](https://mlsanity.com/paper/2402.15806/) (2024)
- [Platypus: A Generalized Specialist Model for Reading Text in Various Forms](https://mlsanity.com/paper/2408.14805/) (2024)
- [Boosting Semi-Supervised Scene Text Recognition via Viewing and Summarizing](https://mlsanity.com/paper/2411.15585/) (2024)
- [MF-LPR2: Multi-Frame License Plate Image Restoration and Recognition using Optical Flow](https://mlsanity.com/paper/2508.14797/) (2025)
- [Looking and Listening: Audio Guided Text Recognition](https://mlsanity.com/paper/2306.03482/) (2023)
- [Efficiently Leveraging Linguistic Priors for Scene Text Spotting](https://mlsanity.com/paper/2402.17134/) (2024)
- [Focus on the Whole Character: Discriminative Character Modeling for Scene Text Recognition](https://mlsanity.com/paper/2407.05562/) (2024)
- [LEGO: Self-Supervised Representation Learning for Scene Text Images](https://mlsanity.com/paper/2408.02036/) (2024)

## Abstract (arXiv)

Scene text recognition (STR) has been an active research topic in computer vision for years. To tackle this challenging problem, numerous innovative methods have been successively proposed and incorporating linguistic knowledge into STR models has recently become a prominent trend. In this work, we first draw inspiration from the recent progress in Vision Transformer (ViT) to construct a conceptually simple yet powerful vision STR model, which is built upon ViT and outperforms previous state-of-the-art models for scene text recognition, including both pure vision models and language-augmented methods. To integrate linguistic knowledge, we further propose a Multi-Granularity Prediction strategy to inject information from the language modality into the model in an implicit way, i.e. , subword representations (BPE and WordPiece) widely-used in NLP are introduced into the output space, in addition to the conventional character level representation, while no independent language model (LM) is adopted. The resultant algorithm (termed MGP-STR) is able to push the performance envelop of STR to an even higher level. Specifically, it achieves an average recognition accuracy of 93.35% on standard benchmarks. Code is available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/MGP-STR.