# Transfer CLIP for Generalizable Image Denoising

arXiv: https://arxiv.org/abs/2403.15132 · HTML: https://mlsanity.com/paper/2403.15132/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 72 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Solving Inverse Problems in Medical Imaging with Score-Based Generative Models](https://mlsanity.com/paper/2111.08005/) (2021)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Noise2Void - Learning Denoising from Single Noisy Images](https://mlsanity.com/paper/1811.10980/) (2018)
- [Denoising Diffusion Models for Plug-and-Play Image Restoration](https://mlsanity.com/paper/2305.08995/) (2023)
- [Noise2Self: Blind Denoising by Self-Supervision](https://mlsanity.com/paper/1901.11365/) (2019)
- [Toward Convolutional Blind Denoising of Real Photographs](https://mlsanity.com/paper/1807.04686/) (2018)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [Generative Diffusion Prior for Unified Image Restoration and Enhancement](https://mlsanity.com/paper/2304.01247/) (2023)
- [SNIPS: Solving Noisy Inverse Problems Stochastically](https://mlsanity.com/paper/2105.14951/) (2021)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [Neighbor2Neighbor: Self-Supervised Denoising from Single Noisy Images](https://mlsanity.com/paper/2101.02824/) (2021)

## Built on by (most cited first, 29 of 29)

- [Post-pre-training for Modality Alignment in Vision-Language Foundation Models](https://mlsanity.com/paper/2504.12717/) (2025)
- [IDF: Iterative Dynamic Filtering Networks for Generalizable Image Denoising](https://mlsanity.com/paper/2508.19649/) (2025)
- [Positive2Negative: Breaking the Information-Lossy Barrier in Self-Supervised Single Image Denoising](https://mlsanity.com/paper/2412.16460/) (2024)
- [Single Domain Generalization for Few-Shot Counting via Universal Representation Matching](https://mlsanity.com/paper/2505.16778/) (2025)
- [Image Denoising Using Green Channel Prior](https://mlsanity.com/paper/2408.05923/) (2024)
- [Learning to Translate Noise for Robust Image Denoising](https://mlsanity.com/paper/2412.04727/) (2024)
- [Beyond Image Prior: Embedding Noise Prior into Conditional Denoising Transformer](https://mlsanity.com/paper/2407.09094/) (2024)
- [Exploring Semantic Feature Discrimination for Perceptual Image Super-Resolution and Opinion-Unaware No-Reference Image Quality Assessment](https://mlsanity.com/paper/2503.19295/) (2025)
- [When Images Look Right and Retrieve Wrong: Coverage-Guided Cross-Scale Re-Indexing for Knowledge-Faithful Generative Perception](https://mlsanity.com/paper/2608.20810/) (2026)
- [BIR-Adapter: A parameter-efficient diffusion adapter for blind image restoration](https://mlsanity.com/paper/2509.06904/) (2025)
- [Selective, Controlled and Domain-Agnostic Unlearning in Pretrained CLIP: A Training- and Data-Free Approach](https://mlsanity.com/paper/2512.14113/) (2025)
- [Unpaired Image Deraining Using Reward-Guided Self-Reinforcement Strategy](https://mlsanity.com/paper/2605.00719/) (2026)
- [Expanding Event Modality Applications through a Robust CLIP-Based Encoder](https://mlsanity.com/paper/2412.03093/) (2024)
- [FM2S: Towards Spatially-Correlated Noise Modeling in Zero-Shot Fluorescence Microscopy Image Denoising](https://mlsanity.com/paper/2412.10031/) (2024)
- [Rethinking High-speed Image Reconstruction Framework with Spike Camera](https://mlsanity.com/paper/2501.04477/) (2025)
- [Prompt Guiding Multi-Scale Adaptive Sparse Representation-driven Network for Low-Dose CT MAR](https://mlsanity.com/paper/2504.19687/) (2025)
- [Towards Robust Image Denoising with Scale Equivariance](https://mlsanity.com/paper/2508.02967/) (2025)
- [Efficient Image Denoising Using Global and Local Circulant Representation](https://mlsanity.com/paper/2508.10307/) (2025)
- [Evaluating Robustness of Vision-Language Models Under Noisy Conditions](https://mlsanity.com/paper/2509.12492/) (2025)
- [Transforming Noise Distributions with Histogram Matching: Towards a Single Denoiser for All](https://mlsanity.com/paper/2510.06757/) (2025)
- [RoGER-SLAM: A Robust Gaussian Splatting SLAM System for Noisy and Low-light Environment Resilience](https://mlsanity.com/paper/2510.22600/) (2025)
- [Self-Supervised Compression and Artifact Correction for Streaming Underwater Imaging Sonar](https://mlsanity.com/paper/2511.13922/) (2025)
- [Image Denoising Using Global and Local Circulant Representation](https://mlsanity.com/paper/2512.23569/) (2025)
- [UCMNet: Uncertainty-Aware Context Memory Network for Under-Display Camera Image Restoration](https://mlsanity.com/paper/2604.00381/) (2026)
- [Beyond Fixed Inference: Quantitative Flow Matching for Adaptive Image Denoising](https://mlsanity.com/paper/2604.02392/) (2026)
- [PISA: A Pseudo-Individual Source-Domain Feature Adaptation Framework for Test-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2608.14142/) (2026)
- [Frozen CLIP Priors for Robust Self-Supervised Poisson Inverse Problems](https://mlsanity.com/paper/2608.20524/) (2026)
- [Pre-Trained Low-Rank Tensor Decomposition for Multi-Dimensional Image Recovery](https://mlsanity.com/paper/2609.12843/) (2026)
- [Beyond Spatial-Domain Supervision: A Relation Constrained Space for Multi-Modal Image Fusion](https://mlsanity.com/paper/2609.38968/) (2026)

## Abstract (arXiv)

Image denoising is a fundamental task in computer vision. While prevailing deep learning-based supervised and self-supervised methods have excelled in eliminating in-distribution noise, their susceptibility to out-of-distribution (OOD) noise remains a significant challenge. The recent emergence of contrastive language-image pre-training (CLIP) model has showcased exceptional capabilities in open-world image recognition and segmentation. Yet, the potential for leveraging CLIP to enhance the robustness of low-level tasks remains largely unexplored. This paper uncovers that certain dense features extracted from the frozen ResNet image encoder of CLIP exhibit distortion-invariant and content-related properties, which are highly desirable for generalizable denoising. Leveraging these properties, we devise an asymmetrical encoder-decoder denoising network, which incorporates dense features including the noisy image and its multi-scale features from the frozen ResNet encoder of CLIP into a learnable image decoder to achieve generalizable denoising. The progressive feature augmentation strategy is further proposed to mitigate feature overfitting and improve the robustness of the learnable decoder. Extensive experiments and comparisons conducted across diverse OOD noises, including synthetic noise, real-world sRGB noise, and low-dose CT image noise, demonstrate the superior generalization ability of our method.