# Scene Text Retrieval via Joint Text Detection and Similarity Learning

arXiv: https://arxiv.org/abs/2104.01552 · HTML: https://mlsanity.com/paper/2104.01552/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 4 Apr 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [Real-time Scene Text Detection with Differentiable Binarization](https://mlsanity.com/paper/1911.08947/) (2019)
- [Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes](https://mlsanity.com/paper/1908.08207/) (2019)
- [ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network](https://mlsanity.com/paper/2002.10200/) (2020)
- [ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition -- RRC-MLT-2019](https://mlsanity.com/paper/1907.00945/) (2019)
- [Reading Text in the Wild with Convolutional Neural Networks](https://mlsanity.com/paper/1412.1842/) (2014)
- [Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting](https://mlsanity.com/paper/2007.09482/) (2020)
- [FOTS: Fast Oriented Text Spotting with a Unified Network](https://mlsanity.com/paper/1801.01671/) (2018)
- [All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting](https://mlsanity.com/paper/1911.09550/) (2019)
- [Towards End-to-end Text Spotting with Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1707.03985/) (2017)
- [An end-to-end TextSpotter with Explicit Alignment and Attention](https://mlsanity.com/paper/1803.03474/) (2018)
- [PHOCNet: A Deep Convolutional Neural Network for Word Spotting in Handwritten Documents](https://mlsanity.com/paper/1604.00187/) (2016)
- [Single Shot Scene Text Retrieval](https://mlsanity.com/paper/1808.09044/) (2018)
- [Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification](https://mlsanity.com/paper/1704.04613/) (2017)
- [Query by String word spotting based on character bi-gram indexing](https://mlsanity.com/paper/1505.07778/) (2015)

## Built on by (most cited first, 15 of 15)

- [Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solution](https://mlsanity.com/paper/2305.07498/) (2023)
- [A Large Cross-Modal Video Retrieval Dataset with Reading Comprehension](https://mlsanity.com/paper/2305.03347/) (2023)
- [ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval](https://mlsanity.com/paper/2203.16778/) (2022)
- [A Bilingual, OpenWorld Video Text Dataset and End-to-end Video Text Spotter with Transformer](https://mlsanity.com/paper/2112.04888/) (2021)
- [Improving Scene Text Image Super-resolution via Dual Prior Modulation Network](https://mlsanity.com/paper/2302.10414/) (2023)
- [A Token-level Text Image Foundation Model for Document Understanding](https://mlsanity.com/paper/2503.02304/) (2025)
- [Knowledge Mining with Scene Text for Fine-Grained Recognition](https://mlsanity.com/paper/2203.14215/) (2022)
- [Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval](https://mlsanity.com/paper/2408.00441/) (2024)
- [Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark](https://mlsanity.com/paper/2512.20174/) (2025)
- [Partial Scene Text Retrieval](https://mlsanity.com/paper/2411.10261/) (2024)
- [Domain Adaptive Scene Text Detection via Subcategorization](https://mlsanity.com/paper/2212.00377/) (2022)
- [FArMARe: a Furniture-Aware Multi-task methodology for Recommending Apartments based on the user interests](https://mlsanity.com/paper/2309.03100/) (2023)
- [A Language-based solution to enable Metaverse Retrieval](https://mlsanity.com/paper/2312.14630/) (2023)
- [MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling](https://mlsanity.com/paper/2506.10609/) (2025)
- [MSLKANet: A Multi-Scale Large Kernel Attention Network for Scene Text Removal](https://mlsanity.com/paper/2211.06565/) (2022)

## Abstract (arXiv)

Scene text retrieval aims to localize and search all text instances from an image gallery, which are the same or similar to a given query text. Such a task is usually realized by matching a query text to the recognized words, outputted by an end-to-end scene text spotter. In this paper, we address this problem by directly learning a cross-modal similarity between a query text and each text instance from natural images. Specifically, we establish an end-to-end trainable network, jointly optimizing the procedures of scene text detection and cross-modal similarity learning. In this way, scene text retrieval can be simply performed by ranking the detected text instances with the learned similarity. Experiments on three benchmark datasets demonstrate our method consistently outperforms the state-of-the-art scene text spotting/retrieval approaches. In particular, the proposed framework of joint detection and similarity learning achieves significantly better performance than separated methods. Code is available at: https://github.com/lanfeng4659/STR-TDSL.