# Weakly-Supervised Text Instance Segmentation

arXiv: https://arxiv.org/abs/2303.10848 · HTML: https://mlsanity.com/paper/2303.10848/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 28 of 28)

- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://mlsanity.com/paper/1507.05717/) (2015)
- [Synthetic Data for Text Localisation in Natural Images](https://mlsanity.com/paper/1604.06646/) (2016)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models](https://mlsanity.com/paper/2109.10282/) (2021)
- [Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations](https://mlsanity.com/paper/1904.05044/) (2019)
- [Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition](https://mlsanity.com/paper/1406.2227/) (2014)
- [Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition](https://mlsanity.com/paper/2103.06495/) (2021)
- [Single-Stage Semantic Segmentation from Image Labels](https://mlsanity.com/paper/2005.08104/) (2020)
- [Total-Text: A Comprehensive Dataset for Scene Text Detection and Recognition](https://mlsanity.com/paper/1710.10400/) (2017)
- [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://mlsanity.com/paper/1811.00751/) (2018)
- [BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation](https://mlsanity.com/paper/2103.08907/) (2021)
- [SVTR: Scene Text Recognition with a Single Visual Model](https://mlsanity.com/paper/2205.00159/) (2022)
- [Vision Transformer for Fast and Efficient Scene Text Recognition](https://mlsanity.com/paper/2105.08582/) (2021)
- [Weakly Supervised Instance Segmentation using Class Peak Response](https://mlsanity.com/paper/1804.00880/) (2018)
- [Look More Than Once: An Accurate Detector for Text of Arbitrary Shapes](https://mlsanity.com/paper/1904.06535/) (2019)
- [Learning Spatial Fusion for Single-Shot Object Detection](https://mlsanity.com/paper/1911.09516/) (2019)
- [Benchmarking Chinese Text Recognition: Datasets, Baselines, and an Empirical Study](https://mlsanity.com/paper/2112.15093/) (2021)
- [Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach](https://mlsanity.com/paper/2011.14021/) (2020)
- [ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors](https://mlsanity.com/paper/1904.03239/) (2019)
- [Object Counting and Instance Segmentation with Image-level Supervision](https://mlsanity.com/paper/1903.02494/) (2019)
- [Text Gestalt: Stroke-Aware Scene Text Image Super-Resolution](https://mlsanity.com/paper/2112.08171/) (2021)
- [Weak Supervision for Generating Pixel-Level Annotations in Scene Text Segmentation](https://mlsanity.com/paper/1911.09026/) (2019)
- [COCO_TS Dataset: Pixel-level Annotations Based on Weak Supervision for Scene Text Segmentation](https://mlsanity.com/paper/1904.00818/) (2019)

## Built on by (most cited first, 4 of 4)

- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentation](https://mlsanity.com/paper/2401.17904/) (2024)
- [EAFormer: Scene Text Segmentation with Edge-Aware Transformers](https://mlsanity.com/paper/2407.17020/) (2024)
- [Learning Attribute-aware Representations for Few-shot Scene Text Segmentation](https://mlsanity.com/paper/2504.11164/) (2025)

## Abstract (arXiv)

Text segmentation is a challenging vision task with many downstream applications. Current text segmentation methods require pixel-level annotations, which are expensive in the cost of human labor and limited in application scenarios. In this paper, we take the first attempt to perform weakly-supervised text instance segmentation by bridging text recognition and text segmentation. The insight is that text recognition methods provide precise attention position of each text instance, and the attention location can feed to both a text adaptive refinement head (TAR) and a text segmentation head. Specifically, the proposed TAR generates pseudo labels by performing two-stage iterative refinement operations on the attention location to fit the accurate boundaries of the corresponding text instance. Meanwhile, the text segmentation head takes the rough attention location to predict segmentation masks which are supervised by the aforementioned pseudo labels. In addition, we design a mask-augmented contrastive learning by treating our segmentation result as an augmented version of the input text image, thus improving the visual representation and further enhancing the performance of both recognition and segmentation. The experimental results demonstrate that the proposed method significantly outperforms weakly-supervised instance segmentation methods on ICDAR13-FST (18.95% improvement) and TextSeg (17.80% improvement) benchmarks.