# Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentation

arXiv: https://arxiv.org/abs/2401.17904 · HTML: https://mlsanity.com/paper/2401.17904/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 94 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Builds on (most cited first, 30 of 69)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [Segment Anything in High Quality](https://mlsanity.com/paper/2306.01567/) (2023)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Character Region Awareness for Text Detection](https://mlsanity.com/paper/1904.01941/) (2019)
- [DocFormer: End-to-End Transformer for Document Understanding](https://mlsanity.com/paper/2106.11539/) (2021)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images](https://mlsanity.com/paper/1601.07140/) (2016)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [Real-time Scene Text Detection with Differentiable Binarization](https://mlsanity.com/paper/1911.08947/) (2019)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)

## Built on by (most cited first, 30 of 30)

- [OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models](https://mlsanity.com/paper/2502.16161/) (2025)
- [LayerD: Decomposing Raster Graphic Designs into Layers](https://mlsanity.com/paper/2509.25134/) (2025)
- [Visual Text Processing: A Comprehensive Review and Unified Evaluation](https://mlsanity.com/paper/2504.21682/) (2025)
- [A Token-level Text Image Foundation Model for Document Understanding](https://mlsanity.com/paper/2503.02304/) (2025)
- [JoyType: A Robust Design for Multilingual Visual Text Creation](https://mlsanity.com/paper/2409.17524/) (2024)
- [Not All Data Are Unlearned Equally](https://mlsanity.com/paper/2504.05058/) (2025)
- [UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesis](https://mlsanity.com/paper/2507.00992/) (2025)
- [Type-R: Automatically Retouching Typos for Text-to-Image Generation](https://mlsanity.com/paper/2411.18159/) (2024)
- [Text-Aware Real-World Image Super-Resolution via Diffusion Model with Joint Segmentation Decoders](https://mlsanity.com/paper/2506.04641/) (2025)
- [Unleashing the Power of Generic Segmentation Models: A Simple Baseline for Infrared Small Target Detection](https://mlsanity.com/paper/2409.04714/) (2024)
- [StyleTextGen: Style-Conditioned Multilingual Scene Text Generation](https://mlsanity.com/paper/2605.14708/) (2026)
- [ConText: Driving In-context Learning for Text Removal and Segmentation](https://mlsanity.com/paper/2506.03799/) (2025)
- [Restore Text First, Enhance Image Later: Two-Stage Scene Text Image Super-Resolution with Glyph Structure Guidance](https://mlsanity.com/paper/2510.21590/) (2025)
- [SFA: Scan, Focus, and Amplify toward Guidance-aware Answering for Video TextVQA](https://mlsanity.com/paper/2511.20190/) (2025)
- [Bharat Scene Text: A Novel Comprehensive Dataset and Benchmark for Indian Language Scene Text Understanding](https://mlsanity.com/paper/2511.23071/) (2025)
- [STRinGS: Selective Text Refinement in Gaussian Splatting](https://mlsanity.com/paper/2512.07230/) (2025)
- [ForCenNet: Foreground-Centric Network for Document Image Rectification](https://mlsanity.com/paper/2507.19804/) (2025)
- [EpiSAM: Character Segmentation in Challenging Stone Inscriptions](https://mlsanity.com/paper/2606.28859/) (2026)
- [ReDesign: Recovering Editable Design Structures from Images via Agentic Decomposition](https://mlsanity.com/paper/2607.25565/) (2026)
- [MG-Gen: Single Image to Motion Graphics Generation](https://mlsanity.com/paper/2504.02361/) (2025)
- [Learning Attribute-aware Representations for Few-shot Scene Text Segmentation](https://mlsanity.com/paper/2504.11164/) (2025)
- [Adapting Segment Anything Model for Power Transmission Corridor Hazard Segmentation](https://mlsanity.com/paper/2505.22105/) (2025)
- [TexTAR : Textual Attribute Recognition in Multi-domain and Multi-lingual Document Images](https://mlsanity.com/paper/2509.13151/) (2025)
- [FontUse: A Data-Centric Approach to Style- and Use-Case-Conditioned In-Image Typography](https://mlsanity.com/paper/2603.06038/) (2026)
- [ET-SAM: Efficient Point Prompt Prediction in SAM for Unified Scene Text Detection and Layout Analysis](https://mlsanity.com/paper/2603.25168/) (2026)
- [VTAgent: Agentic Keyframe Anchoring for Evidence-Aware Video TextVQA](https://mlsanity.com/paper/2605.04870/) (2026)
- [Conversational Retrieval and On-the-Fly Knowledge Modeling of Historical Penitentiary Repression Records](https://mlsanity.com/paper/2607.08459/) (2026)
- [FakeIDet3-DB: Refining Digital Attacks and Patch Extraction for Secure ID Benchmarking](https://mlsanity.com/paper/2607.26641/) (2026)
- [ZMIS-SAM: Segment Anything Model Enhanced with Wavelet Transform for Zooplankton Microscopy Image Instance Segmentation](https://mlsanity.com/paper/2607.27585/) (2026)
- [UniLipi: A Unified Multi-Script OCR for Historical Indic Manuscripts](https://mlsanity.com/paper/2608.28195/) (2026)

## Abstract (arXiv)

The Segment Anything Model (SAM), a profound vision foundation model pretrained on a large-scale dataset, breaks the boundaries of general segmentation and sparks various downstream applications. This paper introduces Hi-SAM, a unified model leveraging SAM for hierarchical text segmentation. Hi-SAM excels in segmentation across four hierarchies, including pixel-level text, word, text-line, and paragraph, while realizing layout analysis as well. Specifically, we first turn SAM into a high-quality pixel-level text segmentation (TS) model through a parameter-efficient fine-tuning approach. We use this TS model to iteratively generate the pixel-level text labels in a semi-automatical manner, unifying labels across the four text hierarchies in the HierText dataset. Subsequently, with these complete labels, we launch the end-to-end trainable Hi-SAM based on the TS architecture with a customized hierarchical mask decoder. During inference, Hi-SAM offers both automatic mask generation (AMG) mode and promptable segmentation (PS) mode. In the AMG mode, Hi-SAM segments pixel-level text foreground masks initially, then samples foreground points for hierarchical text mask generation and achieves layout analysis in passing. As for the PS mode, Hi-SAM provides word, text-line, and paragraph masks with a single point click. Experimental results show the state-of-the-art performance of our TS model: 84.86% fgIOU on Total-Text and 88.96% fgIOU on TextSeg for pixel-level text segmentation. Moreover, compared to the previous specialist for joint hierarchical detection and layout analysis on HierText, Hi-SAM achieves significant improvements: 4.73% PQ and 5.39% F1 on the text-line level, 5.49% PQ and 7.39% F1 on the paragraph level layout analysis, requiring 20× fewer training epochs. The code is available at https://github.com/ymy-k/Hi-SAM.