# Dual-Domain CLIP-Assisted Residual Optimization Perception Model for Metal Artifact Reduction

arXiv: https://arxiv.org/abs/2408.14342 · HTML: https://mlsanity.com/paper/2408.14342/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 14 Aug 2024.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild](https://mlsanity.com/paper/2401.13627/) (2024)
- [DuDoNet: Dual Domain Network for CT Metal Artifact Reduction](https://mlsanity.com/paper/1907.00273/) (2019)
- [Deep Sinogram Completion with Image Prior for Metal Artifact Reduction in CT Images](https://mlsanity.com/paper/2009.07469/) (2020)
- [Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning](https://mlsanity.com/paper/2108.05067/) (2021)
- [InDuDoNet: An Interpretable Dual Domain Network for CT Metal Artifact Reduction](https://mlsanity.com/paper/2109.05298/) (2021)
- [Adaptive Convolutional Dictionary Network for CT Metal Artifact Reduction](https://mlsanity.com/paper/2205.07471/) (2022)
- [Hypernetwork-based Personalized Federated Learning for Multi-Institutional CT Imaging](https://mlsanity.com/paper/2206.03709/) (2022)
- [Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2211.00849/) (2022)
- [ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Data](https://mlsanity.com/paper/2310.05242/) (2023)
- [Unsupervised CT Metal Artifact Reduction by Plugging Diffusion Priors in Dual Domains](https://mlsanity.com/paper/2308.16742/) (2023)
- [DR-GAN: Distribution Regularization for Text-to-Image Generation](https://mlsanity.com/paper/2204.07945/) (2022)
- [Quad-Net: Quad-domain Network for CT Metal Artifact Reduction](https://mlsanity.com/paper/2207.11678/) (2022)
- [Low-dose CT Denoising with Language-engaged Dual-space Alignment](https://mlsanity.com/paper/2403.06128/) (2024)
- [IDOL-Net: An Interactive Dual-Domain Parallel Network for CT Metal Artifact Reduction](https://mlsanity.com/paper/2104.01405/) (2021)
- [PDS-MAR: a fine-grained Projection-Domain Segmentation-based Metal Artifact Reduction method for intraoperative CBCT images with guidewires](https://mlsanity.com/paper/2306.11958/) (2023)

## Built on by (most cited first, 1 of 1)

- [Structure-constrained Language-informed Diffusion Model for Unpaired Low-dose Computed Tomography Angiography Reconstruction](https://mlsanity.com/paper/2601.20304/) (2026)

## Abstract (arXiv)

Metal artifacts in computed tomography (CT) imaging pose significant challenges to accurate clinical diagnosis. The presence of high-density metallic implants results in artifacts that deteriorate image quality, manifesting in the forms of streaking, blurring, or beam hardening effects, etc. Nowadays, various deep learning-based approaches, particularly generative models, have been proposed for metal artifact reduction (MAR). However, these methods have limited perception ability in the diverse morphologies of different metal implants with artifacts, which may generate spurious anatomical structures and exhibit inferior generalization capability. To address the issues, we leverage visual-language model (VLM) to identify these morphological features and introduce them into a dual-domain CLIP-assisted residual optimization perception model (DuDoCROP) for MAR. Specifically, a dual-domain CLIP (DuDoCLIP) is fine-tuned on the image domain and sinogram domain using contrastive learning to extract semantic descriptions from anatomical structures and metal artifacts. Subsequently, a diffusion model is guided by the embeddings of DuDoCLIP, thereby enabling the dual-domain prior generation. Additionally, we design prompt engineering for more precise image-text descriptions that can enhance the model's perception capability. Then, a downstream task is devised for the one-step residual optimization and integration of dual-domain priors, while incorporating raw data fidelity. Ultimately, a new perceptual indicator is proposed to validate the model's perception and generation performance. With the assistance of DuDoCLIP, our DuDoCROP exhibits at least 63.7% higher generalization capability compared to the baseline model. Numerical experiments demonstrate that the proposed method can generate more realistic image structures and outperform other SOTA approaches both qualitatively and quantitatively.