# Semantic Segmentation Prior for Diffusion-Based Real-World Super-Resolution

arXiv: https://arxiv.org/abs/2412.02960 · HTML: https://mlsanity.com/paper/2412.02960/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 15 of its 56 references that are in the map.
- First posted on arXiv 4 Dec 2024.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 15 of 15)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions](https://mlsanity.com/paper/2102.05379/) (2021)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting](https://mlsanity.com/paper/2307.12348/) (2023)
- [DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior](https://mlsanity.com/paper/2308.15070/) (2023)
- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://mlsanity.com/paper/2209.08575/) (2022)
- [DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation](https://mlsanity.com/paper/2410.18666/) (2024)
- [Denoising Diffusion Semantic Segmentation with Mask Prior Modeling](https://mlsanity.com/paper/2306.01721/) (2023)
- [SAM-DiffSR: Structure-Modulated Diffusion Model for Image Super-Resolution](https://mlsanity.com/paper/2402.17133/) (2024)

## Built on by (most cited first, 8 of 8)

- [SRSR: Enhancing Semantic Accuracy in Real-World Image Super-Resolution with Spatially Re-Focused Text-Conditioning](https://mlsanity.com/paper/2510.22534/) (2025)
- [Semantic-Guided Cross-Sensor Super Resolution of Remote Sensing Images: A Gated Dual Conditioning Flow Matching Model](https://mlsanity.com/paper/2510.23816/) (2025)
- [MegaSR: Mining Customized Semantics and Expressive Guidance for Real-World Image Super-Resolution](https://mlsanity.com/paper/2503.08096/) (2025)
- [Nexus-INR: Diverse Knowledge-guided Arbitrary-Scale Multimodal Medical Image Super-Resolution](https://mlsanity.com/paper/2508.03073/) (2025)
- [Adaptive Context Matters: Towards Provable Multi-Modality Guidance for Super-Resolution](https://mlsanity.com/paper/2605.10470/) (2026)
- [Joint Medical Image Enhancement and Segmentation with Diffusion-based Symbiotic Information Interaction](https://mlsanity.com/paper/2607.00058/) (2026)
- [MicroZoom: Structure-Preserving Detail Synthesis at Extreme Scale](https://mlsanity.com/paper/2607.24729/) (2026)
- [ProSR: Semantic-Prototype-Guided Discrete Modeling for Physically Consistent SAR Super-Resolution](https://mlsanity.com/paper/2609.02377/) (2026)

## Abstract (arXiv)

Real-world image super-resolution (Real-ISR) has achieved a remarkable leap by leveraging large-scale text-to-image models, enabling realistic image restoration from given recognition textual prompts. However, these methods sometimes fail to recognize some salient objects, resulting in inaccurate semantic restoration in these regions. Additionally, the same region may have a strong response to more than one prompt and it will lead to semantic ambiguity for image super-resolution. To alleviate the above two issues, in this paper, we propose to consider semantic segmentation as an additional control condition into diffusion-based image super-resolution. Compared to textual prompt conditions, semantic segmentation enables a more comprehensive perception of salient objects within an image by assigning class labels to each pixel. It also mitigates the risks of semantic ambiguities by explicitly allocating objects to their respective spatial regions. In practice, inspired by the fact that image super-resolution and segmentation can benefit each other, we propose SegSR which introduces a dual-diffusion framework to facilitate interaction between the image super-resolution and segmentation diffusion models. Specifically, we develop a Dual-Modality Bridge module to enable updated information flow between these two diffusion models, achieving mutual benefit during the reverse diffusion process. Extensive experiments show that SegSR can generate realistic images while preserving semantic structures more effectively.