# Towards Unified Scene Text Spotting based on Sequence Generation

arXiv: https://arxiv.org/abs/2304.03435 · HTML: https://mlsanity.com/paper/2304.03435/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 43 references that are in the map.
- First posted on arXiv 7 Apr 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Character Region Awareness for Text Detection](https://mlsanity.com/paper/1904.01941/) (2019)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [Real-time Scene Text Detection with Differentiable Binarization](https://mlsanity.com/paper/1911.08947/) (2019)
- [Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes](https://mlsanity.com/paper/1908.08207/) (2019)
- [TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text](https://mlsanity.com/paper/2105.05486/) (2021)
- [ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network](https://mlsanity.com/paper/2002.10200/) (2020)
- [Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition](https://mlsanity.com/paper/1710.10400/) (2017)
- [A Unified Sequence Interface for Vision Tasks](https://mlsanity.com/paper/2206.07669/) (2022)
- [UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling](https://mlsanity.com/paper/2111.12085/) (2021)
- [TextBoxes: A Fast Text Detector with a Single Deep Neural Network](https://mlsanity.com/paper/1611.06779/) (2016)
- [TextBoxes++: A Single-Shot Oriented Scene Text Detector](https://mlsanity.com/paper/1801.02765/) (2018)
- [Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting](https://mlsanity.com/paper/2007.09482/) (2020)
- [FOTS: Fast Oriented Text Spotting with a Unified Network](https://mlsanity.com/paper/1801.01671/) (2018)
- [Towards End-to-End Unified Scene Text Detection and Layout Analysis](https://mlsanity.com/paper/2203.15143/) (2022)
- [ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting](https://mlsanity.com/paper/2105.03620/) (2021)
- [SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognition](https://mlsanity.com/paper/2203.10209/) (2022)
- [Text Spotting Transformers](https://mlsanity.com/paper/2204.01918/) (2022)
- [Rosetta: Large scale system for text detection and recognition in images](https://mlsanity.com/paper/1910.05085/) (2019)
- [MANGO: A Mask Attention Guided One-Stage Scene Text Spotter](https://mlsanity.com/paper/2012.04350/) (2020)
- [PGNet: Real-time Arbitrarily-Shaped Text Spotting with Point Gathering Network](https://mlsanity.com/paper/2104.05458/) (2021)
- [STEFANN: Scene Text Editor using Font Adaptive Neural Network](https://mlsanity.com/paper/1903.01192/) (2019)

## Built on by (most cited first, 27 of 27)

- [OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition](https://mlsanity.com/paper/2403.19128/) (2024)
- [DocVLM: Make Your VLM an Efficient Reader](https://mlsanity.com/paper/2412.08746/) (2024)
- [OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models](https://mlsanity.com/paper/2502.16161/) (2025)
- [DNTextSpotter: Arbitrary-Shaped Scene Text Spotting via Improved Denoising Training](https://mlsanity.com/paper/2408.00355/) (2024)
- [ODM: A Text-Image Further Alignment Pre-training Approach for Scene Text Detection and Spotting](https://mlsanity.com/paper/2403.00303/) (2024)
- [Bridging the Gap Between End-to-End and Two-Step Text Spotting](https://mlsanity.com/paper/2404.04624/) (2024)
- [Hierarchical Text Spotter for Joint Text Spotting and Layout Analysis](https://mlsanity.com/paper/2310.17674/) (2023)
- [InstructOCR: Instruction Boosting Scene Text Spotting](https://mlsanity.com/paper/2412.15523/) (2024)
- [Hyper-Local Deformable Transformers for Text Spotting on Historical Maps](https://mlsanity.com/paper/2506.15010/) (2025)
- [SwinTextSpotter v2: Towards Better Synergy for Scene Text Spotting](https://mlsanity.com/paper/2401.07641/) (2024)
- [Lane2Seq: Towards Unified Lane Detection via Sequence Generation](https://mlsanity.com/paper/2402.17172/) (2024)
- [DocSAM: Unified Document Image Segmentation via Query Decomposition and Heterogeneous Mixed Learning](https://mlsanity.com/paper/2504.04085/) (2025)
- [LRANet++: Low-Rank Approximation Network for Accurate and Efficient Text Spotting](https://mlsanity.com/paper/2511.05818/) (2025)
- [SCOB: Universal Text Understanding via Character-wise Supervised Contrastive Learning with Online Text Rendering for Bridging Domain Gap](https://mlsanity.com/paper/2309.12382/) (2023)
- [CREPE: Coordinate-Aware End-to-End Document Parser](https://mlsanity.com/paper/2405.00260/) (2024)
- [Watermark Text Pattern Spotting in Document Images](https://mlsanity.com/paper/2401.05167/) (2024)
- [Towards Unified Multi-granularity Text Detection with Interactive Attention](https://mlsanity.com/paper/2405.19765/) (2024)
- [Platypus: A Generalized Specialist Model for Reading Text in Various Forms](https://mlsanity.com/paper/2408.14805/) (2024)
- [Conditional Text-to-Image Generation with Reference Guidance](https://mlsanity.com/paper/2411.16713/) (2024)
- [PILOT: A Promptable Interleaved Layout-aware OCR Transformer](https://mlsanity.com/paper/2504.03621/) (2025)
- [Progressive Evolution from Single-Point to Polygon for Scene Text](https://mlsanity.com/paper/2312.13778/) (2023)
- [TextBoost: Boosting Scene Text Fidelity in Ultra-low Bitrate Image Compression](https://mlsanity.com/paper/2603.04115/) (2026)
- [Hybrid Primal Sketch: Combining Analogy, Qualitative Representations, and Computer Vision for Scene Understanding](https://mlsanity.com/paper/2407.04859/) (2024)
- [PositionOCR: Augmenting Positional Awareness in Multi-Modal Models via Hybrid Specialist Integration](https://mlsanity.com/paper/2602.19188/) (2026)
- [TableSeq: Unified Generation of Structure, Content, and Layout](https://mlsanity.com/paper/2604.16070/) (2026)
- [SupGRPO: Enhancing GRPO with Matching-based Online SFT for Text Spotting](https://mlsanity.com/paper/2609.07081/) (2026)
- [MEVL-STP: Multi-Encoder and Vision Language Model for Arbitrarily Shaped Scene Text Spotting](https://mlsanity.com/paper/2609.28857/) (2026)

## Abstract (arXiv)

Sequence generation models have recently made significant progress in unifying various vision tasks. Although some auto-regressive models have demonstrated promising results in end-to-end text spotting, they use specific detection formats while ignoring various text shapes and are limited in the maximum number of text instances that can be detected. To overcome these limitations, we propose a UNIfied scene Text Spotter, called UNITS. Our model unifies various detection formats, including quadrilaterals and polygons, allowing it to detect text in arbitrary shapes. Additionally, we apply starting-point prompting to enable the model to extract texts from an arbitrary starting point, thereby extracting more texts beyond the number of instances it was trained on. Experimental results demonstrate that our method achieves competitive performance compared to state-of-the-art methods. Further analysis shows that UNITS can extract a larger number of texts than it was trained on. We provide the code for our method at https://github.com/clovaai/units.