# Towards Weakly-Supervised Text Spotting using a Multi-Task Transformer

arXiv: https://arxiv.org/abs/2202.05508 · HTML: https://mlsanity.com/paper/2202.05508/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 11 Feb 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes](https://mlsanity.com/paper/1908.08207/) (2019)
- [ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network](https://mlsanity.com/paper/2002.10200/) (2020)
- [Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition](https://mlsanity.com/paper/1710.10400/) (2017)
- [Rethinking Transformer-based Set Prediction for Object Detection](https://mlsanity.com/paper/2011.10881/) (2020)
- [ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)](https://mlsanity.com/paper/1909.07145/) (2019)
- [Reading Text in the Wild with Convolutional Neural Networks](https://mlsanity.com/paper/1412.1842/) (2014)
- [Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting](https://mlsanity.com/paper/2007.09482/) (2020)
- [FOTS: Fast Oriented Text Spotting with a Unified Network](https://mlsanity.com/paper/1801.01671/) (2018)
- [ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting](https://mlsanity.com/paper/2105.03620/) (2021)
- [Convolutional Character Networks](https://mlsanity.com/paper/1910.07954/) (2019)
- [Towards Unconstrained End-to-End Text Spotting](https://mlsanity.com/paper/1908.09231/) (2019)
- [Towards End-to-end Text Spotting with Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1707.03985/) (2017)
- [MANGO: A Mask Attention Guided One-Stage Scene Text Spotter](https://mlsanity.com/paper/2012.04350/) (2020)
- [Character Region Attention For Text Spotting](https://mlsanity.com/paper/2007.09629/) (2020)
- [Optimization for Arbitrary-Oriented Object Detection via Representation Invariance Loss](https://mlsanity.com/paper/2103.11636/) (2021)
- [SEE: Towards Semi-Supervised End-to-End Scene Text Recognition](https://mlsanity.com/paper/1712.05404/) (2017)
- [Text Recognition -- Real World Data and Where to Find Them](https://mlsanity.com/paper/2007.03098/) (2020)

## Built on by (most cited first, 30 of 41)

- [TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document](https://mlsanity.com/paper/2403.04473/) (2024)
- [DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spotting](https://mlsanity.com/paper/2211.10772/) (2022)
- [OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition](https://mlsanity.com/paper/2403.19128/) (2024)
- [SPTS v2: Single-Point Scene Text Spotting](https://mlsanity.com/paper/2301.01635/) (2023)
- [Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentation](https://mlsanity.com/paper/2401.17904/) (2024)
- [ESTextSpotter: Towards Better Scene Text Spotting with Explicit Synergy in Transformer](https://mlsanity.com/paper/2308.10147/) (2023)
- [Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting](https://mlsanity.com/paper/2203.03911/) (2022)
- [Towards Unified Scene Text Spotting based on Sequence Generation](https://mlsanity.com/paper/2304.03435/) (2023)
- [ICDAR 2023 Competition on Hierarchical Text Detection and Recognition](https://mlsanity.com/paper/2305.09750/) (2023)
- [OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models](https://mlsanity.com/paper/2502.16161/) (2025)
- [CLIPTER: Looking at the Bigger Picture in Scene Text Recognition](https://mlsanity.com/paper/2301.07464/) (2023)
- [Towards Models that Can See and Read](https://mlsanity.com/paper/2301.07389/) (2023)
- [GoMatching: A Simple Baseline for Video Text Spotting via Long and Short Term Matching](https://mlsanity.com/paper/2401.07080/) (2024)
- [Out-of-Vocabulary Challenge Report](https://mlsanity.com/paper/2209.06717/) (2022)
- [DNTextSpotter: Arbitrary-Shaped Scene Text Spotting via Improved Denoising Training](https://mlsanity.com/paper/2408.00355/) (2024)
- [Arbitrary Reading Order Scene Text Spotter with Local Semantics Guidance](https://mlsanity.com/paper/2412.10159/) (2024)
- [Turning a CLIP Model into a Scene Text Spotter](https://mlsanity.com/paper/2308.10408/) (2023)
- [Inverse-like Antagonistic Scene Text Spotting via Reading-Order Estimation and Dynamic Sampling](https://mlsanity.com/paper/2401.03637/) (2024)
- [Bridging the Gap Between End-to-End and Two-Step Text Spotting](https://mlsanity.com/paper/2404.04624/) (2024)
- [Hierarchical Text Spotter for Joint Text Spotting and Layout Analysis](https://mlsanity.com/paper/2310.17674/) (2023)
- [DeepSolo++: Let Transformer Decoder with Explicit Points Solo for Multilingual Text Spotting](https://mlsanity.com/paper/2305.19957/) (2023)
- [InstructOCR: Instruction Boosting Scene Text Spotting](https://mlsanity.com/paper/2412.15523/) (2024)
- [Hyper-Local Deformable Transformers for Text Spotting on Historical Maps](https://mlsanity.com/paper/2506.15010/) (2025)
- [Text-Aware Image Restoration with Diffusion Models](https://mlsanity.com/paper/2506.09993/) (2025)
- [SwinTextSpotter v2: Towards Better Synergy for Scene Text Spotting](https://mlsanity.com/paper/2401.07641/) (2024)
- [Weakly supervised information extraction from inscrutable handwritten document images](https://mlsanity.com/paper/2306.06823/) (2023)
- [LRANet++: Low-Rank Approximation Network for Accurate and Efficient Text Spotting](https://mlsanity.com/paper/2511.05818/) (2025)
- [SCOB: Universal Text Understanding via Character-wise Supervised Contrastive Learning with Online Text Rendering for Bridging Domain Gap](https://mlsanity.com/paper/2309.12382/) (2023)
- [WeCromCL: Weakly Supervised Cross-Modality Contrastive Learning for Transcription-only Supervised Text Spotting](https://mlsanity.com/paper/2407.19507/) (2024)
- [Text Detection & Recognition in the Wild for Robot Localization](https://mlsanity.com/paper/2205.08565/) (2022)

## Abstract (arXiv)

Text spotting end-to-end methods have recently gained attention in the literature due to the benefits of jointly optimizing the text detection and recognition components. Existing methods usually have a distinct separation between the detection and recognition branches, requiring exact annotations for the two tasks. We introduce TextTranSpotter (TTS), a transformer-based approach for text spotting and the first text spotting framework which may be trained with both fully- and weakly-supervised settings. By learning a single latent representation per word detection, and using a novel loss function based on the Hungarian loss, our method alleviates the need for expensive localization annotations. Trained with only text transcription annotations on real data, our weakly-supervised method achieves competitive performance with previous state-of-the-art fully-supervised methods. When trained in a fully-supervised manner, TextTranSpotter shows state-of-the-art results on multiple benchmarks.