# A Learnable Variational Model for Joint Multimodal MRI Reconstruction and Synthesis

arXiv: https://arxiv.org/abs/2204.03804 · HTML: https://mlsanity.com/paper/2204.03804/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 8 Apr 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [End-to-End Variational Networks for Accelerated MRI Reconstruction](https://mlsanity.com/paper/2004.06688/) (2020)
- [Image Synthesis in Multi-Contrast MRI with Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1802.01221/) (2018)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis](https://mlsanity.com/paper/2002.05000/) (2020)
- [Deep Residual Learning for Accelerated MRI using Magnitude and Phase Networks](https://mlsanity.com/paper/1804.00432/) (2018)
- [mustGAN: Multi-Stream Generative Adversarial Networks for MR Image Synthesis](https://mlsanity.com/paper/1909.11504/) (2019)
- [Penalty Method for Inversion-Free Deep Bilevel Optimization](https://mlsanity.com/paper/1911.03432/) (2019)
- [Fast OSCAR and OWL Regression via Safe Screening Rules](https://mlsanity.com/paper/2006.16433/) (2020)
- [Model Learning: Primal Dual Networks for Fast MR imaging](https://mlsanity.com/paper/1908.02426/) (2019)
- [Generative Adversarial Networks for Image-to-Image Translation on Multi-Contrast MR Images - A Comparison of CycleGAN and UNIT](https://mlsanity.com/paper/1806.07777/) (2018)
- [An Optimization-Based Meta-Learning Model for MRI Reconstruction with Diverse Dataset](https://mlsanity.com/paper/2110.00715/) (2021)
- [Learnable Descent Algorithm for Nonsmooth Nonconvex Image Reconstruction](https://mlsanity.com/paper/2007.11245/) (2020)
- [An Optimal Control Framework for Joint-channel Parallel MRI Reconstruction without Coil Sensitivities](https://mlsanity.com/paper/2109.09738/) (2021)

## Built on by (most cited first, 16 of 16)

- [Harnessing Earnings Reports for Stock Predictions: A QLoRA-Enhanced LLM Approach](https://mlsanity.com/paper/2408.06634/) (2024)
- [ImageFlowNet: Forecasting Multiscale Image-Level Trajectories of Disease Progression with Irregularly-Sampled Longitudinal Medical Images](https://mlsanity.com/paper/2406.14794/) (2024)
- [Style Transfer: From Stitching to Neural Networks](https://mlsanity.com/paper/2409.00606/) (2024)
- [Multi-task Magnetic Resonance Imaging Reconstruction using Meta-learning](https://mlsanity.com/paper/2403.19966/) (2024)
- [Learned Alternating Minimization Algorithm for Dual-domain Sparse-View CT Reconstruction](https://mlsanity.com/paper/2306.02644/) (2023)
- [A Hybrid Transformer-Mamba Network for Single Image Deraining](https://mlsanity.com/paper/2409.00410/) (2024)
- [Exploiting Partial Common Information Microstructure for Multi-Modal Brain Tumor Segmentation](https://mlsanity.com/paper/2302.02521/) (2023)
- [High Dynamic Range Imaging with Context-aware Transformer](https://mlsanity.com/paper/2304.04416/) (2023)
- [A Brief Overview of Optimization-Based Algorithms for MRI Reconstruction Using Deep Learning](https://mlsanity.com/paper/2406.02626/) (2024)
- [Constrained Probabilistic Mask Learning for Task-specific Undersampled MRI Reconstruction](https://mlsanity.com/paper/2305.16376/) (2023)
- [A Survey on the Application of Generative Adversarial Networks in Cybersecurity: Prospective, Direction and Open Research Scopes](https://mlsanity.com/paper/2407.08839/) (2024)
- [Enhancing Exchange Rate Forecasting with Explainable Deep Learning Models](https://mlsanity.com/paper/2410.19241/) (2024)
- [A Comprehensive Survey on Magnetic Resonance Image Reconstruction](https://mlsanity.com/paper/2503.07097/) (2025)
- [A Learned Proximal Alternating Minimization Algorithm and Its Induced Network for a Class of Two-block Nonconvex and Nonsmooth Optimization](https://mlsanity.com/paper/2411.06333/) (2024)
- [Learning to Borrow Features for Improved Detection of Small Objects in Single-Shot Detectors](https://mlsanity.com/paper/2505.00044/) (2025)
- [Deep Unrolled Meta-Learning for Multi-Coil and Multi-Modality MRI with Adaptive Optimization](https://mlsanity.com/paper/2505.11518/) (2025)

## Abstract (arXiv)

Generating multi-contrasts/modal MRI of the same anatomy enriches diagnostic information but is limited in practice due to excessive data acquisition time. In this paper, we propose a novel deep-learning model for joint reconstruction and synthesis of multi-modal MRI using incomplete k-space data of several source modalities as inputs. The output of our model includes reconstructed images of the source modalities and high-quality image synthesized in the target modality. Our proposed model is formulated as a variational problem that leverages several learnable modality-specific feature extractors and a multimodal synthesis module. We propose a learnable optimization algorithm to solve this model, which induces a multi-phase network whose parameters can be trained using multi-modal MRI data. Moreover, a bilevel-optimization framework is employed for robust parameter training. We demonstrate the effectiveness of our approach using extensive numerical experiments.