# The Power of Context: How Multimodality Improves Image Super-Resolution

arXiv: https://arxiv.org/abs/2503.14503 · HTML: https://mlsanity.com/paper/2503.14503/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 79 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 30 of 67)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Designing a Practical Degradation Model for Deep Blind Image Super-Resolution](https://mlsanity.com/paper/2103.14006/) (2021)

## Built on by (most cited first, 15 of 15)

- [Fine-structure Preserved Real-world Image Super-resolution via Transfer VAE Training](https://mlsanity.com/paper/2507.20291/) (2025)
- [Exploiting Diffusion Prior for Task-driven Image Restoration](https://mlsanity.com/paper/2507.22459/) (2025)
- [Kernel Density Steering: Inference-Time Scaling via Mode Seeking for Image Restoration](https://mlsanity.com/paper/2507.05604/) (2025)
- [From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer Decomposition](https://mlsanity.com/paper/2511.20996/) (2025)
- [TEXTS-Diff: TEXTS-Aware Diffusion Model for Real-World Text Image Super-Resolution](https://mlsanity.com/paper/2601.17340/) (2026)
- [TPGDiff: Hierarchical Triple-Prior Guided Diffusion for Image Restoration](https://mlsanity.com/paper/2601.20306/) (2026)
- [RAGSR: Regional Attention Guided Diffusion for Image Super-Resolution](https://mlsanity.com/paper/2508.16158/) (2025)
- [Reference-Guided Identity Preserving Face Restoration](https://mlsanity.com/paper/2505.21905/) (2025)
- [F2IDiff: Real-world Image Super-resolution using Feature to Image Diffusion Foundation Model](https://mlsanity.com/paper/2512.24473/) (2025)
- [CoDi: Subject-Consistent and Pose-Diverse Text-to-Image Generation](https://mlsanity.com/paper/2507.08396/) (2025)
- [SuperF: Neural Implicit Fields for Multi-Image Super-Resolution](https://mlsanity.com/paper/2512.09115/) (2025)
- [Of All StrIPEs: Investigating Structure-informed Positional Encoding for Efficient Music Generation](https://mlsanity.com/paper/2504.05364/) (2025)
- [Generative Manifold Distillation: Aligning Restoration Trajectories with Natural Image Prior](https://mlsanity.com/paper/2512.11121/) (2025)
- [Are a Thousand Words Better Than a Single Picture? Beyond Images -- A Framework for Multi-Modal Knowledge Graph Dataset Enrichment](https://mlsanity.com/paper/2603.16974/) (2026)
- [MagnifiQ: Patch-aware Text Guided Progressive Upscaling for High-Resolution Image Restoration](https://mlsanity.com/paper/2608.14543/) (2026)

## Abstract (arXiv)

Single-image super-resolution (SISR) remains challenging due to the inherent difficulty of recovering fine-grained details and preserving perceptual quality from low-resolution inputs. Existing methods often rely on limited image priors, leading to suboptimal results. We propose a novel approach that leverages the rich contextual information available in multiple modalities -- including depth, segmentation, edges, and text prompts -- to learn a powerful generative prior for SISR within a diffusion model framework. We introduce a flexible network architecture that effectively fuses multimodal information, accommodating an arbitrary number of input modalities without requiring significant modifications to the diffusion process. Crucially, we mitigate hallucinations, often introduced by text prompts, by using spatial information from other modalities to guide regional text-based conditioning. Each modality's guidance strength can also be controlled independently, allowing steering outputs toward different directions, such as increasing bokeh through depth or adjusting object prominence via segmentation. Extensive experiments demonstrate that our model surpasses state-of-the-art generative SISR methods, achieving superior visual quality and fidelity. See project page at https://mmsr.kfmei.com/.