# Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition

arXiv: https://arxiv.org/abs/2112.12916 · HTML: https://mlsanity.com/paper/2112.12916/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 24 Dec 2021.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Self-Attention Graph Pooling](https://mlsanity.com/paper/1904.08082/) (2019)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias](https://mlsanity.com/paper/2106.03348/) (2021)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://mlsanity.com/paper/2003.12294/) (2020)
- [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://mlsanity.com/paper/1811.00751/) (2018)
- [ICDAR 2019 Robust Reading Challenge on Reading Chinese Text on Signboard](https://mlsanity.com/paper/1912.09641/) (2019)
- [Reading Text in the Wild with Convolutional Neural Networks](https://mlsanity.com/paper/1412.1842/) (2014)
- [Graph-Based Global Reasoning Networks](https://mlsanity.com/paper/1811.12814/) (2018)
- [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://mlsanity.com/paper/2005.10977/) (2020)
- [Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things](https://mlsanity.com/paper/2011.08612/) (2020)
- [Focusing Attention: Towards Accurate Text Recognition in Natural Images](https://mlsanity.com/paper/1709.02054/) (2017)
- [Decoupled Attention Network for Text Recognition](https://mlsanity.com/paper/1912.10205/) (2019)
- [RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition](https://mlsanity.com/paper/2007.07542/) (2020)
- [Deep Relational Reasoning Graph Network for Arbitrary Shape Text Detection](https://mlsanity.com/paper/2003.07493/) (2020)
- [SCATTER: Selective Context Attentional Scene Text Recognizer](https://mlsanity.com/paper/2003.11288/) (2020)
- [TextScanner: Reading Characters in Order for Robust Scene Text Recognition](https://mlsanity.com/paper/1912.12422/) (2019)
- [What If We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labels](https://mlsanity.com/paper/2103.04400/) (2021)
- [Scene Text Recognition from Two-Dimensional Perspective](https://mlsanity.com/paper/1809.06508/) (2018)
- [Linkage Based Face Clustering via Graph Convolution Network](https://mlsanity.com/paper/1903.11306/) (2019)
- [GTC: Guided Training of CTC Towards Efficient and Accurate Scene Text Recognition](https://mlsanity.com/paper/2002.01276/) (2020)
- [On Vocabulary Reliance in Scene Text Recognition](https://mlsanity.com/paper/2005.03959/) (2020)
- [Primitive Representation Learning for Scene Text Recognition](https://mlsanity.com/paper/2105.04286/) (2021)

## Built on by (most cited first, 25 of 25)

- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [DPText-DETR: Towards Better Scene Text Detection with Dynamic Points in Transformer](https://mlsanity.com/paper/2207.04491/) (2022)
- [Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognition](https://mlsanity.com/paper/2208.00438/) (2022)
- [A Survey on Knowledge-Oriented Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10677/) (2025)
- [OCR-IDL: OCR Annotations for Industry Document Library Dataset](https://mlsanity.com/paper/2202.12985/) (2022)
- [Self-supervised Character-to-Character Distillation for Text Recognition](https://mlsanity.com/paper/2211.00288/) (2022)
- [Self-supervised Implicit Glyph Attention for Text Recognition](https://mlsanity.com/paper/2203.03382/) (2022)
- [Levenshtein OCR](https://mlsanity.com/paper/2209.03594/) (2022)
- [Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognition](https://mlsanity.com/paper/2303.04291/) (2023)
- [Context Perception Parallel Decoder for Scene Text Recognition](https://mlsanity.com/paper/2307.12270/) (2023)
- [Instruction-Guided Scene Text Recognition](https://mlsanity.com/paper/2401.17851/) (2024)
- [Scalable Mask Annotation for Video Text Spotting](https://mlsanity.com/paper/2305.01443/) (2023)
- [TPS++: Attention-Enhanced Thin-Plate Spline for Scene Text Recognition](https://mlsanity.com/paper/2305.05322/) (2023)
- [Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition](https://mlsanity.com/paper/2503.18746/) (2025)
- [Decoder Pre-Training with only Text for Scene Text Recognition](https://mlsanity.com/paper/2408.05706/) (2024)
- [Sequential Visual and Semantic Consistency for Semi-supervised Text Recognition](https://mlsanity.com/paper/2402.15806/) (2024)
- [Masked Vision-Language Transformers for Scene Text Recognition](https://mlsanity.com/paper/2211.04785/) (2022)
- [Indian Commercial Truck License Plate Detection and Recognition for Weighbridge Automation](https://mlsanity.com/paper/2211.13194/) (2022)
- [Empowering Agrifood System with Artificial Intelligence: A Survey of the Progress, Challenges and Opportunities](https://mlsanity.com/paper/2305.01899/) (2023)
- [Efficiently Leveraging Linguistic Priors for Scene Text Spotting](https://mlsanity.com/paper/2402.17134/) (2024)
- [Can Large Language Models Unveil the Mysteries? An Exploration of Their Ability to Unlock Information in Complex Scenarios](https://mlsanity.com/paper/2502.19973/) (2025)
- [CMFN: Cross-Modal Fusion Network for Irregular Scene Text Recognition](https://mlsanity.com/paper/2401.10041/) (2024)
- [Open-Vocabulary Scene Text Recognition via Pseudo-Image Labeling and Margin Loss](https://mlsanity.com/paper/2403.07518/) (2024)
- [DOTA: Deformable Optimized Transformer Architecture for End-to-End Text Recognition with Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.04175/) (2025)
- [Joint Low-level and High-level Textual Representation Learning with Multiple Masking Strategies](https://mlsanity.com/paper/2505.06855/) (2025)

## Abstract (arXiv)

Existing Scene Text Recognition (STR) methods typically use a language model to optimize the joint probability of the 1D character sequence predicted by a visual recognition (VR) model, which ignore the 2D spatial context of visual semantics within and between character instances, making them not generalize well to arbitrary shape scene text. To address this issue, we make the first attempt to perform textual reasoning based on visual semantics in this paper. Technically, given the character segmentation maps predicted by a VR model, we construct a subgraph for each instance, where nodes represent the pixels in it and edges are added between nodes based on their spatial similarity. Then, these subgraphs are sequentially connected by their root nodes and merged into a complete graph. Based on this graph, we devise a graph convolutional network for textual reasoning (GTR) by supervising it with a cross-entropy loss. GTR can be easily plugged in representative STR models to improve their performance owing to better textual reasoning. Specifically, we construct our model, namely S-GTR, by paralleling GTR to the language model in a segmentation-based STR baseline, which can effectively exploit the visual-linguistic complementarity via mutual learning. S-GTR sets new state-of-the-art on six challenging STR benchmarks and generalizes well to multi-linguistic datasets. Code is available at https://github.com/adeline-cs/GTR.