# CoactSeg: Learning from Heterogeneous Data for New Multiple Sclerosis Lesion Segmentation

arXiv: https://arxiv.org/abs/2307.04513 · HTML: https://mlsanity.com/paper/2307.04513/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 13 of 13)

- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Automated brain extraction of multi-sequence MRI using artificial neural networks](https://mlsanity.com/paper/1901.11341/) (2019)
- [Mutual Consistency Learning for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2109.09960/) (2021)
- [Semi-supervised Left Atrium Segmentation with Mutual Consistency Training](https://mlsanity.com/paper/2103.02911/) (2021)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [Exploring Smoothness and Class-Separation for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2203.01324/) (2022)
- [Unraveling Privacy Risks of Individual Fairness in Graph Neural Networks](https://mlsanity.com/paper/2301.12951/) (2023)
- [Multiple Sclerosis Lesion Analysis in Brain Magnetic Resonance Images: Techniques and Clinical Applications](https://mlsanity.com/paper/2104.10029/) (2021)
- [Spatially Covariant Lesion Segmentation](https://mlsanity.com/paper/2301.07895/) (2023)
- [Multiple Sclerosis Lesion Activity Segmentation with Attention-Guided Two-Path CNNs](https://mlsanity.com/paper/2008.02001/) (2020)
- [Learn to Ignore: Domain Adaptation for Multi-Site MRI Analysis](https://mlsanity.com/paper/2110.06803/) (2021)
- [4D Deep Learning for Multiple Sclerosis Lesion Activity Segmentation](https://mlsanity.com/paper/2004.09216/) (2020)
- [Team NeuroPoly: Description of the Pipelines for the MICCAI 2021 MS New Lesions Segmentation Challenge](https://mlsanity.com/paper/2109.05409/) (2021)

## Built on by (most cited first, 9 of 9)

- [Diversified and Personalized Multi-rater Medical Image Segmentation](https://mlsanity.com/paper/2403.13417/) (2024)
- [Longitudinal Segmentation of MS Lesions via Temporal Difference Weighting](https://mlsanity.com/paper/2409.13416/) (2024)
- [SegHeD: Segmentation of Heterogeneous Data for Multiple Sclerosis Lesions with Anatomical Constraints](https://mlsanity.com/paper/2410.01766/) (2024)
- [Pre-training Everywhere: Parameter-Efficient Fine-Tuning for Medical Image Analysis via Target Parameter Pre-training](https://mlsanity.com/paper/2408.15011/) (2024)
- [SegHeD+: Segmentation of Heterogeneous Data for Multiple Sclerosis Lesions with Anatomical Constraints and Lesion-aware Augmentation](https://mlsanity.com/paper/2412.10946/) (2024)
- [HyTver: A Novel Loss Function for Longitudinal Multiple Sclerosis Lesion Segmentation](https://mlsanity.com/paper/2508.17639/) (2025)
- [Probabilistic Modeling of Multi-rater Medical Image Segmentation for Diversity and Personalization](https://mlsanity.com/paper/2512.00748/) (2025)
- [Rethinking Evaluation of Multiple Sclerosis (MS) Lesion Segmentation Models](https://mlsanity.com/paper/2605.09666/) (2026)
- [Exploiting Longitudinal Context in Clinician-Verified Interactive Lesion Tracking](https://mlsanity.com/paper/2605.23118/) (2026)

## Abstract (arXiv)

New lesion segmentation is essential to estimate the disease progression and therapeutic effects during multiple sclerosis (MS) clinical treatments. However, the expensive data acquisition and expert annotation restrict the feasibility of applying large-scale deep learning models. Since single-time-point samples with all-lesion labels are relatively easy to collect, exploiting them to train deep models is highly desirable to improve new lesion segmentation. Therefore, we proposed a coaction segmentation (CoactSeg) framework to exploit the heterogeneous data (i.e., new-lesion annotated two-time-point data and all-lesion annotated single-time-point data) for new MS lesion segmentation. The CoactSeg model is designed as a unified model, with the same three inputs (the baseline, follow-up, and their longitudinal brain differences) and the same three outputs (the corresponding all-lesion and new-lesion predictions), no matter which type of heterogeneous data is being used. Moreover, a simple and effective relation regularization is proposed to ensure the longitudinal relations among the three outputs to improve the model learning. Extensive experiments demonstrate that utilizing the heterogeneous data and the proposed longitudinal relation constraint can significantly improve the performance for both new-lesion and all-lesion segmentation tasks. Meanwhile, we also introduce an in-house MS-23v1 dataset, including 38 Oceania single-time-point samples with all-lesion labels. Codes and the dataset are released at https://github.com/ycwu1997/CoactSeg.