# A Bilingual, OpenWorld Video Text Dataset and End-to-end Video Text Spotter with Transformer

arXiv: https://arxiv.org/abs/2112.04888 · HTML: https://mlsanity.com/paper/2112.04888/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 79 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [Real-time Scene Text Detection with Differentiable Binarization](https://mlsanity.com/paper/1911.08947/) (2019)
- [Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes](https://mlsanity.com/paper/1908.08207/) (2019)
- [Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition](https://mlsanity.com/paper/1710.10400/) (2017)
- [Shape Robust Text Detection with Progressive Scale Expansion Network](https://mlsanity.com/paper/1903.12473/) (2019)

## Built on by (most cited first, 26 of 26)

- [MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation](https://mlsanity.com/paper/2411.15262/) (2024)
- [A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension](https://mlsanity.com/paper/2305.03347/) (2023)
- [End-to-End Video Text Spotting with Transformer](https://mlsanity.com/paper/2203.10539/) (2022)
- [MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios](https://mlsanity.com/paper/2505.21333/) (2025)
- [Scalable Mask Annotation for Video Text Spotting](https://mlsanity.com/paper/2305.01443/) (2023)
- [Continual Learning for Image Segmentation with Dynamic Query](https://mlsanity.com/paper/2311.17450/) (2023)
- [GoMatching: A Simple Baseline for Video Text Spotting via Long and Short Term Matching](https://mlsanity.com/paper/2401.07080/) (2024)
- [VidText: Towards Comprehensive Evaluation for Video Text Understanding](https://mlsanity.com/paper/2505.22810/) (2025)
- [Explore Faster Localization Learning For Scene Text Detection](https://mlsanity.com/paper/2207.01342/) (2022)
- [FlowText: Synthesizing Realistic Scene Text Video with Optical Flow Estimation](https://mlsanity.com/paper/2305.03327/) (2023)
- [ICDAR 2023 Video Text Reading Competition for Dense and Small Text](https://mlsanity.com/paper/2304.04376/) (2023)
- [Real-time End-to-End Video Text Spotter with Contrastive Representation Learning](https://mlsanity.com/paper/2207.08417/) (2022)
- [DSText V2: A Comprehensive Video Text Spotting Dataset for Dense and Small Text](https://mlsanity.com/paper/2312.01938/) (2023)
- [Do Current Video LLMs Have Strong OCR Abilities? A Preliminary Study](https://mlsanity.com/paper/2412.20613/) (2024)
- [Scene-Text Grounding for Text-Based Video Question Answering](https://mlsanity.com/paper/2409.14319/) (2024)
- [GoMatching++: Parameter- and Data-Efficient Arbitrary-Shaped Video Text Spotting and Benchmarking](https://mlsanity.com/paper/2505.22228/) (2025)
- [Contrastive Learning of Semantic and Visual Representations for Text Tracking](https://mlsanity.com/paper/2112.14976/) (2021)
- [SteerVTE: Seamless Video Text Editing with Style and Glyph Control](https://mlsanity.com/paper/2606.23254/) (2026)
- [A Systematic Review of NeurIPS Dataset Management Practices](https://mlsanity.com/paper/2411.00266/) (2024)
- [Video text tracking for dense and small text based on pp-yoloe-r and sort algorithm](https://mlsanity.com/paper/2304.00018/) (2023)
- [GloTSFormer: Global Video Text Spotting Transformer](https://mlsanity.com/paper/2401.03694/) (2024)
- [VimTS: A Unified Video and Image Text Spotter for Enhancing the Cross-domain Generalization](https://mlsanity.com/paper/2404.19652/) (2024)
- [LOGO: Video Text Spotting with Language Collaboration and Glyph Perception Model](https://mlsanity.com/paper/2405.19194/) (2024)
- [MSC: A Marine Wildlife Video Dataset with Grounded Segmentation and Clip-Level Captioning](https://mlsanity.com/paper/2508.04549/) (2025)
- [TraRA: Trajectory-level Recognition Aggregation for Video Text Spotting in Urban Surveillance](https://mlsanity.com/paper/2606.07161/) (2026)
- [Back-Tracking from Clarity: Self-Learning to See Text from Afar](https://mlsanity.com/paper/2609.32477/) (2026)

## Abstract (arXiv)

Most existing video text spotting benchmarks focus on evaluating a single language and scenario with limited data. In this work, we introduce a large-scale, Bilingual, Open World Video text benchmark dataset(BOVText). There are four features for BOVText. Firstly, we provide 2,000+ videos with more than 1,750,000+ frames, 25 times larger than the existing largest dataset with incidental text in videos. Secondly, our dataset covers 30+ open categories with a wide selection of various scenarios, e.g., Life Vlog, Driving, Movie, etc. Thirdly, abundant text types annotation (i.e., title, caption or scene text) are provided for the different representational meanings in video. Fourthly, the BOVText provides bilingual text annotation to promote multiple cultures live and communication. Besides, we propose an end-to-end video text spotting framework with Transformer, termed TransVTSpotter, which solves the multi-orient text spotting in video with a simple, but efficient attention-based query-key mechanism. It applies object features from the previous frame as a tracking query for the current frame and introduces a rotation angle prediction to fit the multiorient text instance. On ICDAR2015(video), TransVTSpotter achieves the state-of-the-art performance with 44.1% MOTA, 9 fps. The dataset and code of TransVTSpotter can be found at github:com=weijiawu=BOVText and github:com=weijiawu=TransVTSpotter, respectively.