# MOST: MR reconstruction Optimization for multiple downStream Tasks via continual learning

arXiv: https://arxiv.org/abs/2409.10394 · HTML: https://mlsanity.com/paper/2409.10394/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 51 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Builds on (most cited first, 29 of 29)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [fastMRI: An Open Dataset and Benchmarks for Accelerated MRI](https://mlsanity.com/paper/1811.08839/) (2018)
- [MoDL: Model Based Deep Learning Architecture for Inverse Problems](https://mlsanity.com/paper/1712.02862/) (2017)
- [Learning a Variational Network for Reconstruction of Accelerated MRI Data](https://mlsanity.com/paper/1704.00447/) (2017)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Image reconstruction by domain transform manifold learning](https://mlsanity.com/paper/1704.08841/) (2017)
- [Brain Tumor Segmentation with Deep Neural Networks](https://mlsanity.com/paper/1505.03540/) (2015)
- [Deep learning for undersampled MRI reconstruction](https://mlsanity.com/paper/1709.02576/) (2017)
- [A Deep Cascade of Convolutional Neural Networks for MR Image Reconstruction](https://mlsanity.com/paper/1703.00555/) (2017)
- [Generalized Variational Continual Learning](https://mlsanity.com/paper/2011.12328/) (2020)
- [SKM-TEA: A Dataset for Accelerated MRI Reconstruction with Dense Image Labels for Quantitative Clinical Evaluation](https://mlsanity.com/paper/2203.06823/) (2022)
- [AI-based Reconstruction for Fast MRI -- A Systematic Review and Meta-analysis](https://mlsanity.com/paper/2112.12744/) (2021)
- [Deep learning in radiology: an overview of the concepts and a survey of the state of the art](https://mlsanity.com/paper/1802.08717/) (2018)
- [Quantitative Susceptibility Mapping using Deep Neural Network: QSMnet](https://mlsanity.com/paper/1803.05627/) (2018)
- [Towards continual learning in medical imaging](https://mlsanity.com/paper/1811.02496/) (2018)
- [Joint CS-MRI Reconstruction and Segmentation with a Unified Deep Network](https://mlsanity.com/paper/1805.02165/) (2018)
- [Importance Driven Continual Learning for Segmentation Across Domains](https://mlsanity.com/paper/2005.00079/) (2020)
- [The International Workshop on Osteoarthritis Imaging Knee MRI Segmentation Challenge: A Multi-Institute Evaluation and Analysis Framework on a Standardized Dataset](https://mlsanity.com/paper/2004.14003/) (2020)
- [Deep reinforcement learning-designed radiofrequency waveform in MRI](https://mlsanity.com/paper/2105.03061/) (2021)
- [A Segmentation-aware Deep Fusion Network for Compressed Sensing MRI](https://mlsanity.com/paper/1804.01210/) (2018)
- [One Network to Solve Them All: A Sequential Multi-Task Joint Learning Network Framework for MR Imaging Pipeline](https://mlsanity.com/paper/2105.06653/) (2021)
- [Artificial neural network for myelin water imaging](https://mlsanity.com/paper/1904.12522/) (2019)
- [Learning Task-Specific Strategies for Accelerated MRI](https://mlsanity.com/paper/2304.12507/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Comprehensive Survey on Magnetic Resonance Image Reconstruction](https://mlsanity.com/paper/2503.07097/) (2025)
- [Exploiting Intermediate Reconstructions in Optical Coherence Tomography for Test-Time Adaption of Medical Image Segmentation](https://mlsanity.com/paper/2603.05041/) (2026)
- [Vision-Language Model-Guided Deep Unrolling Enables Personalized, Fast MRI](https://mlsanity.com/paper/2604.06849/) (2026)

## Abstract (arXiv)

Deep learning-based Magnetic Resonance (MR) reconstruction methods have focused on generating high-quality images but often overlook the impact on downstream tasks (e.g., segmentation) that utilize the reconstructed images. Cascading separately trained reconstruction network and downstream task network has been shown to introduce performance degradation due to error propagation and domain gaps between training datasets. To mitigate this issue, downstream task-oriented reconstruction optimization has been proposed for a single downstream task. Expanding this optimization to multi-task scenarios is not straightforward. In this work, we extended this optimization to sequentially introduced multiple downstream tasks and demonstrated that a single MR reconstruction network can be optimized for multiple downstream tasks by deploying continual learning (MOST). MOST integrated techniques from replay-based continual learning and image-guided loss to overcome catastrophic forgetting. Comparative experiments demonstrated that MOST outperformed a reconstruction network without finetuning, a reconstruction network with na"ive finetuning, and conventional continual learning methods. The source code is available at: https://github.com/SNU-LIST/MOST.