# Anatomy-guided Pathology Segmentation

arXiv: https://arxiv.org/abs/2407.05844 · HTML: https://mlsanity.com/paper/2407.05844/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 8 Jul 2024.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017)
- [Boundary IoU: Improving Object-Centric Image Segmentation Evaluation](https://mlsanity.com/paper/2103.16562/) (2021)
- [Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation](https://mlsanity.com/paper/1705.08302/) (2017)
- [Depth-aware CNN for RGB-D Segmentation](https://mlsanity.com/paper/1803.06791/) (2018)
- [LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching](https://mlsanity.com/paper/2306.11925/) (2023)
- [ChestX-Det10: Chest X-ray Dataset on Detection of Thoracic Abnormalities](https://mlsanity.com/paper/2006.10550/) (2020)
- [Accurate Fine-Grained Segmentation of Human Anatomy in Radiographs via Volumetric Pseudo-Labeling](https://mlsanity.com/paper/2306.03934/) (2023)
- [Towards Unifying Anatomy Segmentation: Automated Generation of a Full-body CT Dataset via Knowledge Aggregation and Anatomical Guidelines](https://mlsanity.com/paper/2307.13375/) (2023)
- [Shape-Aware Complementary-Task Learning for Multi-Organ Segmentation](https://mlsanity.com/paper/1908.05099/) (2019)
- [Anatomy-Driven Pathology Detection on Chest X-rays](https://mlsanity.com/paper/2309.02578/) (2023)
- [ShuffleSeg: Real-time Semantic Segmentation Network](https://mlsanity.com/paper/1803.03816/) (2018)
- [AG-CRC: Anatomy-Guided Colorectal Cancer Segmentation in CT with Imperfect Anatomical Knowledge](https://mlsanity.com/paper/2310.04677/) (2023)

## Built on by (most cited first, 7 of 7)

- [CF-Seg: Counterfactuals meet Segmentation](https://mlsanity.com/paper/2506.16213/) (2025)
- [Autopet III challenge: Incorporating anatomical knowledge into nnUNet for lesion segmentation in PET/CT](https://mlsanity.com/paper/2409.12155/) (2024)
- [REN: Anatomically-Informed Mixture-of-Experts for Interstitial Lung Disease Diagnosis](https://mlsanity.com/paper/2510.04923/) (2025)
- [BackSplit: The Importance of Sub-dividing the Background in Biomedical Lesion Segmentation](https://mlsanity.com/paper/2511.19394/) (2025)
- [Invisible Yet Detected: PelFANet with Attention-Guided Anatomical Fusion for Pelvic Fracture Diagnosis](https://mlsanity.com/paper/2509.13873/) (2025)
- [GRASPing Anatomy to Improve Pathology Segmentation](https://mlsanity.com/paper/2508.03374/) (2025)
- [DE-KAN: A Kolmogorov Arnold Network with Dual Encoder for accurate 2D Teeth Segmentation](https://mlsanity.com/paper/2511.18533/) (2025)

## Abstract (arXiv)

Pathological structures in medical images are typically deviations from the expected anatomy of a patient. While clinicians consider this interplay between anatomy and pathology, recent deep learning algorithms specialize in recognizing either one of the two, rarely considering the patient's body from such a joint perspective. In this paper, we develop a generalist segmentation model that combines anatomical and pathological information, aiming to enhance the segmentation accuracy of pathological features. Our Anatomy-Pathology Exchange (APEx) training utilizes a query-based segmentation transformer which decodes a joint feature space into query-representations for human anatomy and interleaves them via a mixing strategy into the pathology-decoder for anatomy-informed pathology predictions. In doing so, we are able to report the best results across the board on FDG-PET-CT and Chest X-Ray pathology segmentation tasks with a margin of up to 3.3% as compared to strong baseline methods. Code and models will be publicly available at github.com/alexanderjaus/APEx.