# Vision-Language Controlled Deep Unfolding for Joint Medical Image Restoration and Segmentation

arXiv: https://arxiv.org/abs/2601.23103 · HTML: https://mlsanity.com/paper/2601.23103/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Learning Deep CNN Denoiser Prior for Image Restoration](https://mlsanity.com/paper/1704.03264/) (2017)
- [Deep Generalized Unfolding Networks for Image Restoration](https://mlsanity.com/paper/2204.13348/) (2022)
- [PromptIR: Prompting for All-in-One Blind Image Restoration](https://mlsanity.com/paper/2306.13090/) (2023)
- [Low-Rank Few-Shot Adaptation of Vision-Language Models](https://mlsanity.com/paper/2405.18541/) (2024)
- [Towards Realistic Data Generation for Real-World Super-Resolution](https://mlsanity.com/paper/2406.07255/) (2024)
- [Vision-Language Gradient Descent-driven All-in-One Deep Unfolding Networks](https://mlsanity.com/paper/2503.16930/) (2025)
- [All-In-One Medical Image Restoration via Task-Adaptive Routing](https://mlsanity.com/paper/2405.19769/) (2024)
- [Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline](https://mlsanity.com/paper/2411.12814/) (2024)
- [MedSAM3: Delving into Segment Anything with Medical Concepts](https://mlsanity.com/paper/2511.19046/) (2025)
- [DenoiSeg: Joint Denoising and Segmentation](https://mlsanity.com/paper/2005.02987/) (2020)
- [OTLRM: Orthogonal Learning-based Low-Rank Metric for Multi-Dimensional Inverse Problems](https://mlsanity.com/paper/2412.11165/) (2024)
- [U-Shape Mamba: State Space Model for faster diffusion](https://mlsanity.com/paper/2504.13499/) (2025)
- [CENet: Context Enhancement Network for Medical Image Segmentation](https://mlsanity.com/paper/2505.18423/) (2025)
- [All-in-One Medical Image Restoration with Latent Diffusion-Enhanced Vector-Quantized Codebook Prior](https://mlsanity.com/paper/2507.19874/) (2025)
- [TAT: Task-Adaptive Transformer for All-in-One Medical Image Restoration](https://mlsanity.com/paper/2512.14550/) (2025)
- [Self is the Best Learner: CT-free Ultra-Low-Dose PET Organ Segmentation via Collaborating Denoising and Segmentation Learning](https://mlsanity.com/paper/2503.03786/) (2025)
- [Fine-Tuning MedGemma for Clinical Captioning to Enhance Multimodal RAG over Malaysia CPGs](https://mlsanity.com/paper/2510.15418/) (2025)

## Abstract (arXiv)

We propose VL-DUN, a principled framework for joint All-in-One Medical Image Restoration and Segmentation (AiOMIRS) that bridges the gap between low-level signal recovery and high-level semantic understanding. While standard pipelines treat these tasks in isolation, our core insight is that they are fundamentally synergistic: restoration provides clean anatomical structures to improve segmentation, while semantic priors regularize the restoration process. VL-DUN resolves the sub-optimality of sequential processing through two primary innovations. (1) We formulate AiOMIRS as a unified optimization problem, deriving an interpretable joint unfolding mechanism where restoration and segmentation are mathematically coupled for mutual refinement. (2) We introduce a frequency-aware Mamba mechanism to capture long-range dependencies for global segmentation while preserving the high-frequency textures necessary for restoration. This allows for efficient global context modeling with linear complexity, effectively mitigating the spectral bias of standard architectures. As a pioneering work in the AiOMIRS task, VL-DUN establishes a new state-of-the-art across multi-modal benchmarks, improving PSNR by 0.92 dB and the Dice coefficient by 9.76%. Our results demonstrate that joint collaborative learning offers a superior, more robust solution for complex clinical workflows compared to isolated task processing. The codes are provided in https://github.com/cipi666/VLDUN.