# End-to-End Video Text Spotting with Transformer

arXiv: https://arxiv.org/abs/2203.10539 · HTML: https://mlsanity.com/paper/2203.10539/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 81 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [Character Region Awareness for Text Detection](https://mlsanity.com/paper/1904.01941/) (2019)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [Arbitrary-Oriented Scene Text Detection via Rotation Proposals](https://mlsanity.com/paper/1703.01086/) (2017)
- [Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes](https://mlsanity.com/paper/1908.08207/) (2019)
- [ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network](https://mlsanity.com/paper/2002.10200/) (2020)
- [Shape Robust Text Detection with Progressive Scale Expansion Network](https://mlsanity.com/paper/1903.12473/) (2019)
- [FOTS: Fast Oriented Text Spotting with a Unified Network](https://mlsanity.com/paper/1801.01671/) (2018)

## Built on by (most cited first, 22 of 22)

- [Tracking Objects as Pixel-wise Distributions](https://mlsanity.com/paper/2207.05518/) (2022)
- [MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation](https://mlsanity.com/paper/2411.15262/) (2024)
- [A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension](https://mlsanity.com/paper/2305.03347/) (2023)
- [PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Sampling](https://mlsanity.com/paper/2410.05970/) (2024)
- [Scalable Mask Annotation for Video Text Spotting](https://mlsanity.com/paper/2305.01443/) (2023)
- [Continual Learning for Image Segmentation with Dynamic Query](https://mlsanity.com/paper/2311.17450/) (2023)
- [GoMatching: A Simple Baseline for Video Text Spotting via Long and Short Term Matching](https://mlsanity.com/paper/2401.07080/) (2024)
- [Explore Faster Localization Learning For Scene Text Detection](https://mlsanity.com/paper/2207.01342/) (2022)
- [FlowText: Synthesizing Realistic Scene Text Video with Optical Flow Estimation](https://mlsanity.com/paper/2305.03327/) (2023)
- [ICDAR 2023 Video Text Reading Competition for Dense and Small Text](https://mlsanity.com/paper/2304.04376/) (2023)
- [Real-time End-to-End Video Text Spotter with Contrastive Representation Learning](https://mlsanity.com/paper/2207.08417/) (2022)
- [DSText V2: A Comprehensive Video Text Spotting Dataset for Dense and Small Text](https://mlsanity.com/paper/2312.01938/) (2023)
- [GoMatching++: Parameter- and Data-Efficient Arbitrary-Shaped Video Text Spotting and Benchmarking](https://mlsanity.com/paper/2505.22228/) (2025)
- [Dual-Stream Knowledge-Preserving Hashing for Unsupervised Video Retrieval](https://mlsanity.com/paper/2310.08009/) (2023)
- [SFA: Scan, Focus, and Amplify toward Guidance-aware Answering for Video TextVQA](https://mlsanity.com/paper/2511.20190/) (2025)
- [Video text tracking for dense and small text based on pp-yoloe-r and sort algorithm](https://mlsanity.com/paper/2304.00018/) (2023)
- [GloTSFormer: Global Video Text Spotting Transformer](https://mlsanity.com/paper/2401.03694/) (2024)
- [VimTS: A Unified Video and Image Text Spotter for Enhancing the Cross-domain Generalization](https://mlsanity.com/paper/2404.19652/) (2024)
- [LOGO: Video Text Spotting with Language Collaboration and Glyph Perception Model](https://mlsanity.com/paper/2405.19194/) (2024)
- [TraRA: Trajectory-level Recognition Aggregation for Video Text Spotting in Urban Surveillance](https://mlsanity.com/paper/2606.07161/) (2026)
- [Back-Tracking from Clarity: Self-Learning to See Text from Afar](https://mlsanity.com/paper/2609.32477/) (2026)
- [TextVBench: Evaluating Text Rendering and Preservation in Video Generation](https://mlsanity.com/paper/2610.13412/) (2026)

## Abstract (arXiv)

Recent video text spotting methods usually require the three-staged pipeline, i.e., detecting text in individual images, recognizing localized text, tracking text streams with post-processing to generate final results. These methods typically follow the tracking-by-match paradigm and develop sophisticated pipelines. In this paper, rooted in Transformer sequence modeling, we propose a simple, but effective end-to-end video text DEtection, Tracking, and Recognition framework (TransDETR). TransDETR mainly includes two advantages: 1) Different from the explicit match paradigm in the adjacent frame, TransDETR tracks and recognizes each text implicitly by the different query termed text query over long-range temporal sequence (more than 7 frames). 2) TransDETR is the first end-to-end trainable video text spotting framework, which simultaneously addresses the three sub-tasks (e.g., text detection, tracking, recognition). Extensive experiments in four video text datasets (i.e.,ICDAR2013 Video, ICDAR2015 Video, Minetto, and YouTube Video Text) are conducted to demonstrate that TransDETR achieves state-of-the-art performance with up to around 8.0% improvements on video text spotting tasks. The code of TransDETR can be found at https://github.com/weijiawu/TransDETR.