# Detection and Rectification of Arbitrary Shaped Scene Texts by using Text Keypoints and Links

arXiv: https://arxiv.org/abs/2103.00785 · HTML: https://mlsanity.com/paper/2103.00785/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 67 references that are in the map.
- First posted on arXiv 1 Mar 2021.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [Character Region Awareness for Text Detection](https://mlsanity.com/paper/1904.01941/) (2019)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [Real-time Scene Text Detection with Differentiable Binarization](https://mlsanity.com/paper/1911.08947/) (2019)
- [Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes](https://mlsanity.com/paper/1908.08207/) (2019)
- [Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition](https://mlsanity.com/paper/1710.10400/) (2017)
- [Shape Robust Text Detection with Progressive Scale Expansion Network](https://mlsanity.com/paper/1903.12473/) (2019)
- [ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT)](https://mlsanity.com/paper/1909.07145/) (2019)
- [Robust Scene Text Recognition with Automatic Rectification](https://mlsanity.com/paper/1603.03915/) (2016)
- [TextBoxes++: A Single-Shot Oriented Scene Text Detector](https://mlsanity.com/paper/1801.02765/) (2018)
- [TextSnake: A Flexible Representation for Detecting Text of Arbitrary Shapes](https://mlsanity.com/paper/1807.01544/) (2018)
- [Efficient and Accurate Arbitrary-Shaped Text Detection with Pixel Aggregation Network](https://mlsanity.com/paper/1908.05900/) (2019)
- [FOTS: Fast Oriented Text Spotting with a Unified Network](https://mlsanity.com/paper/1801.01671/) (2018)
- [Rotation-Sensitive Regression for Oriented Scene Text Detection](https://mlsanity.com/paper/1803.05265/) (2018)
- [Detecting Text in Natural Image with Connectionist Text Proposal Network](https://mlsanity.com/paper/1609.03605/) (2016)
- [Detecting Curve Text in the Wild: New Dataset and New Solution](https://mlsanity.com/paper/1712.02170/) (2017)
- [Detecting Oriented Text in Natural Images by Linking Segments](https://mlsanity.com/paper/1703.06520/) (2017)
- [ESIR: End-to-end Scene Text Recognition via Iterative Image Rectification](https://mlsanity.com/paper/1812.05824/) (2018)
- [ContourNet: Taking a Further Step toward Accurate Arbitrary-shaped Scene Text Detection](https://mlsanity.com/paper/2004.04940/) (2020)
- [Look More Than Once: An Accurate Detector for Text of Arbitrary Shapes](https://mlsanity.com/paper/1904.06535/) (2019)
- [All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting](https://mlsanity.com/paper/1911.09550/) (2019)
- [PixelLink: Detecting Scene Text via Instance Segmentation](https://mlsanity.com/paper/1801.01315/) (2018)

## Built on by (most cited first, 5 of 5)

- [Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting](https://mlsanity.com/paper/2203.03911/) (2022)
- [Contextual Text Block Detection towards Scene Text Understanding](https://mlsanity.com/paper/2207.12955/) (2022)
- [Image Projective Transformation Rectification with Synthetic Data for Smartphone-captured Chest X-ray Photos Classification](https://mlsanity.com/paper/2210.05954/) (2022)
- [Domain Adaptive Scene Text Detection via Subcategorization](https://mlsanity.com/paper/2212.00377/) (2022)
- [Dynamic Relation Transformer for Contextual Text Block Detection](https://mlsanity.com/paper/2401.09232/) (2024)

## Abstract (arXiv)

Detection and recognition of scene texts of arbitrary shapes remain a grand challenge due to the super-rich text shape variation in text line orientations, lengths, curvatures, etc. This paper presents a mask-guided multi-task network that detects and rectifies scene texts of arbitrary shapes reliably. Three types of keypoints are detected which specify the centre line and so the shape of text instances accurately. In addition, four types of keypoint links are detected of which the horizontal links associate the detected keypoints of each text instance and the vertical links predict a pair of landmark points (for each keypoint) along the upper and lower text boundary, respectively. Scene texts can be located and rectified by linking up the associated landmark points (giving localization polygon boxes) and transforming the polygon boxes via thin plate spline, respectively. Extensive experiments over several public datasets show that the use of text keypoints is tolerant to the variation in text orientations, lengths, and curvatures, and it achieves superior scene text detection and rectification performance as compared with state-of-the-art methods.