# The Devil is in Fine-tuning and Long-tailed Problems:A New Benchmark for Scene Text Detection

arXiv: https://arxiv.org/abs/2505.15649 · HTML: https://mlsanity.com/paper/2505.15649/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 49 references that are in the map.
- First posted on arXiv 21 May 2025.

## Builds on (most cited first, 30 of 40)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [Real-time Scene Text Detection with Differentiable Binarization](https://mlsanity.com/paper/1911.08947/) (2019)
- [TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text](https://mlsanity.com/paper/2105.05486/) (2021)
- [ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network](https://mlsanity.com/paper/2002.10200/) (2020)
- [Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition](https://mlsanity.com/paper/1710.10400/) (2017)
- [ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)](https://mlsanity.com/paper/1909.07145/) (2019)
- [ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019](https://mlsanity.com/paper/1907.00945/) (2019)
- [Reading Text in the Wild with Convolutional Neural Networks](https://mlsanity.com/paper/1412.1842/) (2014)
- [ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVT](https://mlsanity.com/paper/1909.07741/) (2019)
- [TextBoxes: A Fast Text Detector with a Single Deep Neural Network](https://mlsanity.com/paper/1611.06779/) (2016)
- [TextBoxes++: A Single-Shot Oriented Scene Text Detector](https://mlsanity.com/paper/1801.02765/) (2018)
- [TextSnake: A Flexible Representation for Detecting Text of Arbitrary Shapes](https://mlsanity.com/paper/1807.01544/) (2018)
- [Efficient and Accurate Arbitrary-Shaped Text Detection with Pixel Aggregation Network](https://mlsanity.com/paper/1908.05900/) (2019)
- [Towards End-to-End Unified Scene Text Detection and Layout Analysis](https://mlsanity.com/paper/2203.15143/) (2022)
- [ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting](https://mlsanity.com/paper/2105.03620/) (2021)
- [Detecting Curve Text in the Wild: New Dataset and New Solution](https://mlsanity.com/paper/1712.02170/) (2017)
- [Few Could Be Better Than All: Feature Sampling and Grouping for Scene Text Detection](https://mlsanity.com/paper/2203.15221/) (2022)
- [DPText-DETR: Towards Better Scene Text Detection with Dynamic Points in Transformer](https://mlsanity.com/paper/2207.04491/) (2022)
- [Revisiting Scene Text Recognition: A Data Perspective](https://mlsanity.com/paper/2307.08723/) (2023)
- [Turning a CLIP Model into a Scene Text Detector](https://mlsanity.com/paper/2302.14338/) (2023)
- [GA-DAN: Geometry-Aware Domain Adaptation Network for Scene Text Detection and Recognition](https://mlsanity.com/paper/1907.09653/) (2019)
- [TextRay: Contour-based Geometric Modeling for Arbitrary-shaped Scene Text Detection](https://mlsanity.com/paper/2008.04851/) (2020)
- [Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting](https://mlsanity.com/paper/2203.03911/) (2022)

## Built on by (most cited first, 3 of 3)

- [StyleTextGen: Style-Conditioned Multilingual Scene Text Generation](https://mlsanity.com/paper/2605.14708/) (2026)
- [Class-Agnostic Region-of-Interest Matching in Document Images](https://mlsanity.com/paper/2506.21055/) (2025)
- [Orthogonal Knowledge Refreshing for Domain-Incremental Object Detection](https://mlsanity.com/paper/2607.17340/) (2026)

## Abstract (arXiv)

Scene text detection has seen the emergence of high-performing methods that excel on academic benchmarks. However, these detectors often fail to replicate such success in real-world scenarios. We uncover two key factors contributing to this discrepancy through extensive experiments. First, a textitFine-tuning Gap, where models leverage textitDataset-Specific Optimization (DSO) paradigm for one domain at the cost of reduced effectiveness in others, leads to inflated performances on academic benchmarks. Second, the suboptimal performance in practical settings is primarily attributed to the long-tailed distribution of texts, where detectors struggle with rare and complex categories as artistic or overlapped text. Given that the DSO paradigm might undermine the generalization ability of models, we advocate for a textitJoint-Dataset Learning (JDL) protocol to alleviate the Fine-tuning Gap. Additionally, an error analysis is conducted to identify three major categories and 13 subcategories of challenges in long-tailed scene text, upon which we propose a Long-Tailed Benchmark (LTB). LTB facilitates a comprehensive evaluation of ability to handle a diverse range of long-tailed challenges. We further introduce MAEDet, a self-supervised learning-based method, as a strong baseline for LTB. The code is available at https://github.com/pd162/LTB.