# Scene Text Recognition with Semantics

arXiv: https://arxiv.org/abs/2210.10836 · HTML: https://mlsanity.com/paper/2210.10836/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis](https://mlsanity.com/paper/1904.01906/) (2019)
- [TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text](https://mlsanity.com/paper/2105.05486/) (2021)
- [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://mlsanity.com/paper/1811.00751/) (2018)
- [Robust Scene Text Recognition with Automatic Rectification](https://mlsanity.com/paper/1603.03915/) (2016)
- [Scene Text Detection and Recognition: The Deep Learning Era](https://mlsanity.com/paper/1811.04256/) (2018)
- [Focusing Attention: Towards Accurate Text Recognition in Natural Images](https://mlsanity.com/paper/1709.02054/) (2017)
- [VinVL: Revisiting Visual Representations in Vision-Language Models](https://mlsanity.com/paper/2101.00529/) (2021)
- [Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1312.6082/) (2013)
- [Rosetta: Large scale system for text detection and recognition in images](https://mlsanity.com/paper/1910.05085/) (2019)
- [SCATTER: Selective Context Attentional Scene Text Recognizer](https://mlsanity.com/paper/2003.11288/) (2020)
- [MASTER: Multi-Aspect Non-local Network for Scene Text Recognition](https://mlsanity.com/paper/1910.02562/) (2019)
- [A Survey on Optical Character Recognition System](https://mlsanity.com/paper/1710.05703/) (2017)
- [STN-OCR: A single Neural Network for Text Detection and Text Recognition](https://mlsanity.com/paper/1707.08831/) (2017)

## Abstract (arXiv)

Scene Text Recognition (STR) models have achieved high performance in recent years on benchmark datasets where text images are presented with minimal noise. Traditional STR recognition pipelines take a cropped image as sole input and attempt to identify the characters present. This infrastructure can fail in instances where the input image is noisy or the text is partially obscured. This paper proposes using semantic information from the greater scene to contextualise predictions. We generate semantic vectors using object tags and fuse this information into a transformer-based architecture. The results demonstrate that our multimodal approach yields higher performance than traditional benchmark models, particularly on noisy instances.