# STEP -- Towards Structured Scene-Text Spotting

arXiv: https://arxiv.org/abs/2309.02356 · HTML: https://mlsanity.com/paper/2309.02356/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 5 Sep 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text](https://mlsanity.com/paper/2105.05486/) (2021)
- [ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network](https://mlsanity.com/paper/2002.10200/) (2020)
- [Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition](https://mlsanity.com/paper/1710.10400/) (2017)
- [ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019](https://mlsanity.com/paper/1907.00945/) (2019)
- [TextBoxes: A Fast Text Detector with a Single Deep Neural Network](https://mlsanity.com/paper/1611.06779/) (2016)
- [Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting](https://mlsanity.com/paper/2007.09482/) (2020)
- [FOTS: Fast Oriented Text Spotting with a Unified Network](https://mlsanity.com/paper/1801.01671/) (2018)
- [Towards End-to-End Unified Scene Text Detection and Layout Analysis](https://mlsanity.com/paper/2203.15143/) (2022)
- [ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting](https://mlsanity.com/paper/2105.03620/) (2021)
- [Detecting Curve Text in the Wild: New Dataset and New Solution](https://mlsanity.com/paper/1712.02170/) (2017)
- [SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognition](https://mlsanity.com/paper/2203.10209/) (2022)
- [Text Spotting Transformers](https://mlsanity.com/paper/2204.01918/) (2022)
- [A Robust Real-Time Automatic License Plate Recognition Based on the YOLO Detector](https://mlsanity.com/paper/1802.09567/) (2018)
- [Open Images V5 Text Annotation and Yet Another Mask Text Spotter](https://mlsanity.com/paper/2106.12326/) (2021)
- [SPTS: Single-Point Text Spotting](https://mlsanity.com/paper/2112.07917/) (2021)
- [Towards Weakly-Supervised Text Spotting using a Multi-Task Transformer](https://mlsanity.com/paper/2202.05508/) (2022)
- [On Vocabulary Reliance in Scene Text Recognition](https://mlsanity.com/paper/2005.03959/) (2020)
- [Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries](https://mlsanity.com/paper/2105.02170/) (2021)
- [RoadText-1K: Text Detection & Recognition Dataset for Driving Videos](https://mlsanity.com/paper/2005.09496/) (2020)
- [Out-of-Vocabulary Challenge Report](https://mlsanity.com/paper/2209.06717/) (2022)
- [DEER: Detection-agnostic End-to-End Recognizer for Scene Text Spotting](https://mlsanity.com/paper/2203.05122/) (2022)
- [Improving Structured Text Recognition with Regular Expression Biasing](https://mlsanity.com/paper/2111.06738/) (2021)

## Built on by (most cited first, 2 of 2)

- [InstructOCR: Instruction Boosting Scene Text Spotting](https://mlsanity.com/paper/2412.15523/) (2024)
- [FastTextSpotter: A High-Efficiency Transformer for Multilingual Scene Text Spotting](https://mlsanity.com/paper/2408.14998/) (2024)

## Abstract (arXiv)

We introduce the structured scene-text spotting task, which requires a scene-text OCR system to spot text in the wild according to a query regular expression. Contrary to generic scene text OCR, structured scene-text spotting seeks to dynamically condition both scene text detection and recognition on user-provided regular expressions. To tackle this task, we propose the Structured TExt sPotter (STEP), a model that exploits the provided text structure to guide the OCR process. STEP is able to deal with regular expressions that contain spaces and it is not bound to detection at the word-level granularity. Our approach enables accurate zero-shot structured text spotting in a wide variety of real-world reading scenarios and is solely trained on publicly available data. To demonstrate the effectiveness of our approach, we introduce a new challenging test dataset that contains several types of out-of-vocabulary structured text, reflecting important reading applications of fields such as prices, dates, serial numbers, license plates etc. We demonstrate that STEP can provide specialised OCR performance on demand in all tested scenarios.