# HATs: Hierarchical Adaptive Taxonomy Segmentation for Panoramic Pathology Image Analysis

arXiv: https://arxiv.org/abs/2407.00596 · HTML: https://mlsanity.com/paper/2407.00596/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 30 Jun 2024.

## Builds on (most cited first, 15 of 15)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything](https://mlsanity.com/paper/2312.00863/) (2023)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning](https://mlsanity.com/paper/2307.00290/) (2023)
- [Iterative annotation to ease neural network training: Specialized machine learning in medical image analysis](https://mlsanity.com/paper/1812.07509/) (2018)
- [Omni-Seg: A Scale-aware Dynamic Network for Renal Pathological Image Segmentation](https://mlsanity.com/paper/2206.13632/) (2022)
- [PrPSeg: Universal Proposition Learning for Panoramic Renal Pathology Segmentation](https://mlsanity.com/paper/2402.19286/) (2024)
- [Democratizing Pathological Image Segmentation with Lay Annotators via Molecular-empowered Learning](https://mlsanity.com/paper/2306.00047/) (2023)
- [Spatial Pathomics Toolkit for Quantitative Analysis of Podocyte Nuclei with Histology and Spatial Transcriptomics Data in Renal Pathology](https://mlsanity.com/paper/2308.06288/) (2023)

## Built on by (most cited first, 9 of 9)

- [Evaluating Cell AI Foundation Models in Kidney Pathology with Human-in-the-Loop Enrichment](https://mlsanity.com/paper/2411.00078/) (2024)
- [PFPs: Prompt-guided Flexible Pathological Segmentation for Diverse Potential Outcomes Using Large Vision and Language Models](https://mlsanity.com/paper/2407.09979/) (2024)
- [BackSplit: The Importance of Sub-dividing the Background in Biomedical Lesion Segmentation](https://mlsanity.com/paper/2511.19394/) (2025)
- [ZeroReg3D: A Zero-shot Registration Pipeline for 3D Consecutive Histopathology Image Reconstruction](https://mlsanity.com/paper/2506.21923/) (2025)
- [M^3-GloDets: Multi-Region and Multi-Scale Analysis of Fine-Grained Diseased Glomerular Detection](https://mlsanity.com/paper/2508.17666/) (2025)
- [CASC-AI: Consensus-aware Self-corrective Learning for Noise Cell Segmentation](https://mlsanity.com/paper/2502.07302/) (2025)
- [Label tree semantic losses for rich multi-class medical image segmentation](https://mlsanity.com/paper/2507.15777/) (2025)
- [MORI-Seg: Learning Morphological Geometry for Instance Segmentation without Instance Annotations](https://mlsanity.com/paper/2605.28261/) (2026)
- [Seeing Abnormal from Normal: Glomerular Abnormality in Representations of Normal Renal Morphology](https://mlsanity.com/paper/2609.19444/) (2026)

## Abstract (arXiv)

Panoramic image segmentation in computational pathology presents a remarkable challenge due to the morphologically complex and variably scaled anatomy. For instance, the intricate organization in kidney pathology spans multiple layers, from regions like the cortex and medulla to functional units such as glomeruli, tubules, and vessels, down to various cell types. In this paper, we propose a novel Hierarchical Adaptive Taxonomy Segmentation (HATs) method, which is designed to thoroughly segment panoramic views of kidney structures by leveraging detailed anatomical insights. Our approach entails (1) the innovative HATs technique which translates spatial relationships among 15 distinct object classes into a versatile "plug-and-play" loss function that spans across regions, functional units, and cells, (2) the incorporation of anatomical hierarchies and scale considerations into a unified simple matrix representation for all panoramic entities, (3) the adoption of the latest AI foundation model (EfficientSAM) as a feature extraction tool to boost the model's adaptability, yet eliminating the need for manual prompt generation in conventional segment anything model (SAM). Experimental findings demonstrate that the HATs method offers an efficient and effective strategy for integrating clinical insights and imaging precedents into a unified segmentation model across more than 15 categories. The official implementation is publicly available at https://github.com/hrlblab/HATs.