# Multimodal Interactive Lung Lesion Segmentation: A Framework for Annotating PET/CT Images based on Physiological and Anatomical Cues

arXiv: https://arxiv.org/abs/2301.09914 · HTML: https://mlsanity.com/paper/2301.09914/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 23 references that are in the map.
- First posted on arXiv 24 Jan 2023.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning](https://mlsanity.com/paper/1710.04043/) (2017)
- [DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation](https://mlsanity.com/paper/1707.00652/) (2017)
- [MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning](https://mlsanity.com/paper/2104.12166/) (2021)
- [MONAI Label: A framework for AI-assisted Interactive Labeling of 3D Medical Images](https://mlsanity.com/paper/2203.12362/) (2022)
- [Going to Extremes: Weakly Supervised Medical Image Segmentation](https://mlsanity.com/paper/2009.11988/) (2020)
- [Uncertainty-Guided Efficient Interactive Refinement of Fetal Brain Segmentation from Stacks of MRI Slices](https://mlsanity.com/paper/2007.00833/) (2020)

## Built on by (most cited first, 6 of 6)

- [Deep Interactive Segmentation of Medical Images: A Systematic Review and Taxonomy](https://mlsanity.com/paper/2311.13964/) (2023)
- [Mirror U-Net: Marrying Multimodal Fission with Multi-task Learning for Semantic Segmentation in Medical Imaging](https://mlsanity.com/paper/2303.07126/) (2023)
- [Guiding the Guidance: A Comparative Analysis of User Guidance Signals for Interactive Segmentation of Volumetric Images](https://mlsanity.com/paper/2303.06942/) (2023)
- [Sliding Window FastEdit: A Framework for Lesion Annotation in Whole-body PET Images](https://mlsanity.com/paper/2311.14482/) (2023)
- [AutoPET Challenge 2023: Sliding Window-based Optimization of U-Net](https://mlsanity.com/paper/2309.12114/) (2023)
- [Rethinking Annotator Simulation: Realistic Evaluation of Whole-Body PET Lesion Interactive Segmentation Methods](https://mlsanity.com/paper/2404.01816/) (2024)

## Abstract (arXiv)

Recently, deep learning enabled the accurate segmentation of various diseases in medical imaging. These performances, however, typically demand large amounts of manual voxel annotations. This tedious process for volumetric data becomes more complex when not all required information is available in a single imaging domain as is the case for PET/CT data. We propose a multimodal interactive segmentation framework that mitigates these issues by combining anatomical and physiological cues from PET/CT data. Our framework utilizes the geodesic distance transform to represent the user annotations and we implement a novel ellipsoid-based user simulation scheme during training. We further propose two annotation interfaces and conduct a user study to estimate their usability. We evaluated our model on the in-domain validation dataset and an unseen PET/CT dataset. We make our code publicly available: https://github.com/verena-hallitschke/pet-ct-annotate.