# Distilling Semantic Priors from SAM to Efficient Image Restoration Models

arXiv: https://arxiv.org/abs/2403.16368 · HTML: https://mlsanity.com/paper/2403.16368/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Builds on (most cited first, 11 of 11)

- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Simple Baselines for Image Restoration](https://mlsanity.com/paper/2204.04676/) (2022)
- [Toward Convolutional Blind Denoising of Real Photographs](https://mlsanity.com/paper/1807.04686/) (2018)
- [SinDDM: A Single Image Denoising Diffusion Model](https://mlsanity.com/paper/2211.16582/) (2022)
- [When Image Denoising Meets High-Level Vision Tasks: A Deep Learning Approach](https://mlsanity.com/paper/1706.04284/) (2017)
- [EfficientDeRain: Learning Pixel-wise Dilation Filtering for High-Efficiency Single-Image Deraining](https://mlsanity.com/paper/2009.09238/) (2020)
- [IDR: Self-Supervised Image Denoising via Iterative Data Refinement](https://mlsanity.com/paper/2111.14358/) (2021)
- [Real-World Deep Local Motion Deblurring](https://mlsanity.com/paper/2204.08179/) (2022)
- [Exploiting Semantics for Face Image Deblurring](https://mlsanity.com/paper/2001.06822/) (2020)
- [Let Segment Anything Help Image Dehaze](https://mlsanity.com/paper/2306.15870/) (2023)
- [SAM-Deblur: Let Segment Anything Boost Image Deblurring](https://mlsanity.com/paper/2309.02270/) (2023)

## Built on by (most cited first, 25 of 25)

- [TAB: Unified Benchmarking of Time Series Anomaly Detection Methods](https://mlsanity.com/paper/2506.18046/) (2025)
- [Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent](https://mlsanity.com/paper/2501.01230/) (2025)
- [Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models](https://mlsanity.com/paper/2403.17359/) (2024)
- [From Zero to Detail: Deconstructing Ultra-High-Definition Image Restoration from Progressive Spectral Perspective](https://mlsanity.com/paper/2503.13165/) (2025)
- [GIM: A Million-scale Benchmark for Generative Image Manipulation Detection and Localization](https://mlsanity.com/paper/2406.16531/) (2024)
- [AWRaCLe: All-Weather Image Restoration using Visual In-Context Learning](https://mlsanity.com/paper/2409.00263/) (2024)
- [ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices](https://mlsanity.com/paper/2506.03737/) (2025)
- [Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models](https://mlsanity.com/paper/2411.08466/) (2024)
- [Rethinking Pseudo-Label Guided Learning for Weakly Supervised Temporal Action Localization from the Perspective of Noise Correction](https://mlsanity.com/paper/2501.11124/) (2025)
- [K2VAE: A Koopman-Kalman Enhanced Variational AutoEncoder for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2505.23017/) (2025)
- [ASTGI: Adaptive Spatio-Temporal Graph Interactions for Irregular Multivariate Time Series Forecasting](https://mlsanity.com/paper/2509.23313/) (2025)
- [From Zero to Detail: A Progressive Spectral Decoupling Paradigm for UHD Image Restoration with New Benchmark](https://mlsanity.com/paper/2604.15654/) (2026)
- [Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration](https://mlsanity.com/paper/2412.08939/) (2024)
- [Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era](https://mlsanity.com/paper/2511.06024/) (2025)
- [Knowledge Distillation with Multi-granularity Mixture of Priors for Image Super-Resolution](https://mlsanity.com/paper/2404.02573/) (2024)
- [An End-to-End Real-World Camera Imaging Pipeline](https://mlsanity.com/paper/2411.10773/) (2024)
- [Boosting HDR Image Reconstruction via Semantic Knowledge Transfer](https://mlsanity.com/paper/2503.15361/) (2025)
- [Enhancing Features in Long-tailed Data Using Large Vision Model](https://mlsanity.com/paper/2504.10852/) (2025)
- [CLIP-AE: CLIP-assisted Cross-view Audio-Visual Enhancement for Unsupervised Temporal Action Localization](https://mlsanity.com/paper/2505.23524/) (2025)
- [Real-World Adverse Weather Image Restoration via Dual-Level Reinforcement Learning with High-Quality Cold Start](https://mlsanity.com/paper/2511.05095/) (2025)
- [InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective](https://mlsanity.com/paper/2505.21920/) (2025)
- [FCL-COD: Weakly Supervised Camouflaged Object Detection with Frequency-aware and Contrastive Learning](https://mlsanity.com/paper/2603.22969/) (2026)
- [PixIE: Prompted Pixel-Space Low-Light Image Enhancement](https://mlsanity.com/paper/2605.23531/) (2026)
- [Rapid Bone Scintigraphy Enhancement via Semantic Prior Distillation from Segment Anything Model](https://mlsanity.com/paper/2503.02321/) (2025)
- [Semantic-E2VID: a Semantic-Enriched Paradigm for Event-to-Video Reconstruction](https://mlsanity.com/paper/2510.17347/) (2025)

## Abstract (arXiv)

In image restoration (IR), leveraging semantic priors from segmentation models has been a common approach to improve performance. The recent segment anything model (SAM) has emerged as a powerful tool for extracting advanced semantic priors to enhance IR tasks. However, the computational cost of SAM is prohibitive for IR, compared to existing smaller IR models. The incorporation of SAM for extracting semantic priors considerably hampers the model inference efficiency. To address this issue, we propose a general framework to distill SAM's semantic knowledge to boost exiting IR models without interfering with their inference process. Specifically, our proposed framework consists of the semantic priors fusion (SPF) scheme and the semantic priors distillation (SPD) scheme. SPF fuses two kinds of information between the restored image predicted by the original IR model and the semantic mask predicted by SAM for the refined restored image. SPD leverages a self-distillation manner to distill the fused semantic priors to boost the performance of original IR models. Additionally, we design a semantic-guided relation (SGR) module for SPD, which ensures semantic feature representation space consistency to fully distill the priors. We demonstrate the effectiveness of our framework across multiple IR models and tasks, including deraining, deblurring, and denoising.