# Scene Text Recognition with Permuted Autoregressive Sequence Models

arXiv: https://arxiv.org/abs/2207.06966 · HTML: https://mlsanity.com/paper/2207.06966/

## Key facts

- Cited by 149 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 84 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text](https://mlsanity.com/paper/2105.05486/) (2021)
- [ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)](https://mlsanity.com/paper/1909.07145/) (2019)
- [ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019](https://mlsanity.com/paper/1907.00945/) (2019)
- [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://mlsanity.com/paper/2003.12294/) (2020)
- [ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboard](https://mlsanity.com/paper/1912.09641/) (2019)
- [ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVT](https://mlsanity.com/paper/1909.07741/) (2019)
- [Robust Scene Text Recognition with Automatic Rectification](https://mlsanity.com/paper/1603.03915/) (2016)
- [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://mlsanity.com/paper/2005.10977/) (2020)

## Built on by (most cited first, 30 of 149)

- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark](https://mlsanity.com/paper/2306.06687/) (2023)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering](https://mlsanity.com/paper/2311.16465/) (2023)
- [UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models](https://mlsanity.com/paper/2312.04884/) (2023)
- [Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer](https://mlsanity.com/paper/2311.13120/) (2023)
- [From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models](https://mlsanity.com/paper/2403.12027/) (2024)
- [Revisiting Scene Text Recognition: A Data Perspective](https://mlsanity.com/paper/2307.08723/) (2023)
- [DTrOCR: Decoder-only Transformer for Optical Character Recognition](https://mlsanity.com/paper/2308.15996/) (2023)
- [FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions](https://mlsanity.com/paper/2305.17718/) (2023)
- [CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model](https://mlsanity.com/paper/2305.14014/) (2023)
- [Self-supervised Character-to-Character Distillation for Text Recognition](https://mlsanity.com/paper/2211.00288/) (2022)
- [CoLLaVO: Crayon Large Language and Vision mOdel](https://mlsanity.com/paper/2402.11248/) (2024)
- [SVTRv2: CTC Beats Encoder-Decoder Models in Scene Text Recognition](https://mlsanity.com/paper/2411.15858/) (2024)
- [ICDAR 2023 Competition on Hierarchical Text Detection and Recognition](https://mlsanity.com/paper/2305.09750/) (2023)
- [Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognition](https://mlsanity.com/paper/2303.04291/) (2023)
- [Context Perception Parallel Decoder for Scene Text Recognition](https://mlsanity.com/paper/2307.12270/) (2023)
- [DreamText: High Fidelity Scene Text Synthesis](https://mlsanity.com/paper/2405.14701/) (2024)
- [Instruction-Guided Scene Text Recognition](https://mlsanity.com/paper/2401.17851/) (2024)
- [CLIPTER: Looking at the Bigger Picture in Scene Text Recognition](https://mlsanity.com/paper/2301.07464/) (2023)
- [Scene Text Image Super-resolution based on Text-conditional Diffusion Models](https://mlsanity.com/paper/2311.09759/) (2023)
- [Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition](https://mlsanity.com/paper/2310.04999/) (2023)
- [Layout Agnostic Scene Text Image Synthesis with Diffusion Models](https://mlsanity.com/paper/2406.01062/) (2024)
- [LISTER: Neighbor Decoding for Length-Insensitive Scene Text Recognition](https://mlsanity.com/paper/2308.12774/) (2023)
- [TokBench: Evaluating Your Visual Tokenizer before Visual Generation](https://mlsanity.com/paper/2505.18142/) (2025)
- [UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters](https://mlsanity.com/paper/2512.21095/) (2025)
- [IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognition](https://mlsanity.com/paper/2312.11923/) (2023)

## Abstract (arXiv)

Context-aware STR methods typically use internal autoregressive (AR) language models (LM). Inherent limitations of AR models motivated two-stage methods which employ an external LM. The conditional independence of the external LM on the input image may cause it to erroneously rectify correct predictions, leading to significant inefficiencies. Our method, PARSeq, learns an ensemble of internal AR LMs with shared weights using Permutation Language Modeling. It unifies context-free non-AR and context-aware AR inference, and iterative refinement using bidirectional context. Using synthetic training data, PARSeq achieves state-of-the-art (SOTA) results in STR benchmarks (91.9% accuracy) and more challenging datasets. It establishes new SOTA results (96.0% accuracy) when trained on real data. PARSeq is optimal on accuracy vs parameter count, FLOPS, and latency because of its simple, unified structure and parallel token processing. Due to its extensive use of attention, it is robust on arbitrarily-oriented text which is common in real-world images. Code, pretrained weights, and data are available at: https://github.com/baudm/parseq.