# Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach

arXiv: https://arxiv.org/abs/2011.14021 · HTML: https://mlsanity.com/paper/2011.14021/

## Key facts

- Cited by 45 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 27 Nov 2020.

## Built on by (most cited first, 30 of 45)

- [TextDiffuser: Diffusion Models as Text Painters](https://mlsanity.com/paper/2305.10855/) (2023)
- [Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering](https://mlsanity.com/paper/2403.09622/) (2024)
- [UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models](https://mlsanity.com/paper/2312.04884/) (2023)
- [UltraSR: Spatial Encoding is a Missing Key for Implicit Image Function-based Arbitrary-Scale Super-Resolution](https://mlsanity.com/paper/2103.12716/) (2021)
- [TextStyleBrush: Transfer of Text Aesthetics from a Single Example](https://mlsanity.com/paper/2106.08385/) (2021)
- [Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognition](https://mlsanity.com/paper/2207.00193/) (2022)
- [Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognition](https://mlsanity.com/paper/2208.00438/) (2022)
- [Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentation](https://mlsanity.com/paper/2401.17904/) (2024)
- [Self-supervised Character-to-Character Distillation for Text Recognition](https://mlsanity.com/paper/2211.00288/) (2022)
- [Aesthetic Text Logo Synthesis via Content-aware Layout Inferring](https://mlsanity.com/paper/2204.02701/) (2022)
- [Don't Forget Me: Accurate Background Recovery for Text Removal via Modeling Local-Global Context](https://mlsanity.com/paper/2207.10273/) (2022)
- [Visual Text Processing: A Comprehensive Review and Unified Evaluation](https://mlsanity.com/paper/2504.21682/) (2025)
- [DreamText: High Fidelity Scene Text Synthesis](https://mlsanity.com/paper/2405.14701/) (2024)
- [Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation](https://mlsanity.com/paper/2403.16422/) (2024)
- [FETNet: Feature Erasing and Transferring Network for Scene Text Removal](https://mlsanity.com/paper/2306.09593/) (2023)
- [UPOCR: Towards Unified Pixel-Level OCR Interface](https://mlsanity.com/paper/2312.02694/) (2023)
- [Class-Aware Mask-Guided Feature Refinement for Scene Text Recognition](https://mlsanity.com/paper/2402.13643/) (2024)
- [ControlText: Unlocking Controllable Fonts in Multilingual Text Rendering without Font Annotations](https://mlsanity.com/paper/2502.10999/) (2025)
- [Bridging Synthetic and Real Worlds for Pre-training Scene Text Detectors](https://mlsanity.com/paper/2312.05286/) (2023)
- [Kinetic Typography Diffusion Model](https://mlsanity.com/paper/2407.10476/) (2024)
- [How Control Information Influences Multilingual Text Image Generation and Editing?](https://mlsanity.com/paper/2407.11502/) (2024)
- [Visual Text Meets Low-level Vision: A Comprehensive Survey on Visual Text Processing](https://mlsanity.com/paper/2402.03082/) (2024)
- [Text-Aware Real-World Image Super-Resolution via Diffusion Model with Joint Segmentation Decoders](https://mlsanity.com/paper/2506.04641/) (2025)
- [A Scene-Text Synthesis Engine Achieved Through Learning from Decomposed Real-World Data](https://mlsanity.com/paper/2209.02397/) (2022)
- [WAS: Dataset and Methods for Artistic Text Segmentation](https://mlsanity.com/paper/2408.00106/) (2024)
- [Vision as Unified Multimodal Generation](https://mlsanity.com/paper/2607.06560/) (2026)
- [Weakly-Supervised Text Instance Segmentation](https://mlsanity.com/paper/2303.10848/) (2023)
- [EAFormer: Scene Text Segmentation with Edge-Aware Transformers](https://mlsanity.com/paper/2407.17020/) (2024)
- [UM-Text: A Unified Multimodal Model for Image Understanding and Visual Text Editing](https://mlsanity.com/paper/2601.08321/) (2026)
- [MonkeyOCRv2: A Visual-Text Foundation Model for Document AI](https://mlsanity.com/paper/2607.11562/) (2026)

## Abstract (arXiv)

Text segmentation is a prerequisite in many real-world text-related tasks, e.g., text style transfer, and scene text removal. However, facing the lack of high-quality datasets and dedicated investigations, this critical prerequisite has been left as an assumption in many works, and has been largely overlooked by current research. To bridge this gap, we proposed TextSeg, a large-scale fine-annotated text dataset with six types of annotations: word- and character-wise bounding polygons, masks and transcriptions. We also introduce Text Refinement Network (TexRNet), a novel text segmentation approach that adapts to the unique properties of text, e.g. non-convex boundary, diverse texture, etc., which often impose burdens on traditional segmentation models. In our TexRNet, we propose text specific network designs to address such challenges, including key features pooling and attention-based similarity checking. We also introduce trimap and discriminator losses that show significant improvement on text segmentation. Extensive experiments are carried out on both our TextSeg dataset and other existing datasets. We demonstrate that TexRNet consistently improves text segmentation performance by nearly 2% compared to other state-of-the-art segmentation methods. Our dataset and code will be made available at https://github.com/SHI-Labs/Rethinking-Text-Segmentation.