# Frozen CLIP Priors for Robust Self-Supervised Poisson Inverse Problems

arXiv: https://arxiv.org/abs/2608.20524 · HTML: https://mlsanity.com/paper/2608.20524/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 20 Aug 2026.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [MoDL: Model Based Deep Learning Architecture for Inverse Problems](https://mlsanity.com/paper/1712.02862/) (2017)
- [Denoising Diffusion Models for Plug-and-Play Image Restoration](https://mlsanity.com/paper/2305.08995/) (2023)
- [The Little Engine that Could: Regularization by Denoising (RED)](https://mlsanity.com/paper/1611.02862/) (2016)
- [Frozen CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2208.03550/) (2022)
- [Gradient Step Denoiser for convergent Plug-and-Play](https://mlsanity.com/paper/2110.03220/) (2021)
- [Equivariant Imaging: Learning Beyond the Range Space](https://mlsanity.com/paper/2103.14756/) (2021)
- [A Poisson-Gaussian Denoising Dataset with Real Fluorescence Microscopy Images](https://mlsanity.com/paper/1812.10366/) (2018)
- [Robust Equivariant Imaging: a fully unsupervised framework for learning to image from noisy and partial measurements](https://mlsanity.com/paper/2111.12855/) (2021)
- [Transfer CLIP for Generalizable Image Denoising](https://mlsanity.com/paper/2403.15132/) (2024)
- [Unsupervised Learning From Incomplete Measurements for Inverse Problems](https://mlsanity.com/paper/2201.12151/) (2022)
- [UNSURE: self-supervised learning with Unknown Noise level and Stein's Unbiased Risk Estimate](https://mlsanity.com/paper/2409.01985/) (2024)
- [Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2406.11189/) (2024)
- [Scale-Equivariant Imaging: Self-Supervised Learning for Image Super-Resolution and Deblurring](https://mlsanity.com/paper/2312.11232/) (2023)
- [Generalized Recorrupted-to-Recorrupted: Self-Supervised Learning Beyond Gaussian Noise](https://mlsanity.com/paper/2412.04648/) (2024)
- [Self-Supervised Learning from Noisy and Incomplete Data](https://mlsanity.com/paper/2601.03244/) (2026)

## Abstract (arXiv)

Self-supervised learning for imaging inverse problems is increasingly important in photon-limited settings, where acquiring clean ground truth is impractical and reconstruction must remain stable under dataset and acquisition shifts. This challenge is amplified under Poisson noise, whose signal-dependent statistics interact with sampling operators (e.g., CFA mosaicing). Meanwhile, foundation vision encoders trained at web scale offer distortion-invariant, content-related representations that generalize well across domains, suggesting a promising route to build priors that transfer beyond the training distribution without expensive fine-tuning. This paper proposes an ADMM-inspired unrolled plug-and-play solver for Poisson inverse problems that decouples a closed-form data-consistency update from a parameter-efficient prior. The prior is implemented as a lightweight decoder operating on frozen CLIP RN50 dense multi-scale features, adapting foundation representations with less trainable parameters. For self-supervision, the method integrates GR2R measurement-domain re-corruption with an Equivariant Imaging regularizer via virtual acquisitions. Experiments on Poisson CFA demosaicing and deblurring show competitive quality, improved robustness under shifts, and self-supervised performance approaching supervised training.