# Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling

arXiv: https://arxiv.org/abs/2108.05301 · HTML: https://mlsanity.com/paper/2108.05301/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 11 Aug 2021.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Designing a Practical Degradation Model for Deep Blind Image Super-Resolution](https://mlsanity.com/paper/2103.14006/) (2021)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design](https://mlsanity.com/paper/1902.00275/) (2019)
- [Deep Unfolding Network for Image Super-Resolution](https://mlsanity.com/paper/2003.10428/) (2020)
- [Analyzing Inverse Problems with Invertible Neural Networks](https://mlsanity.com/paper/1808.04730/) (2018)
- [SRFlow: Learning the Super-Resolution Space with Normalizing Flow](https://mlsanity.com/paper/2006.14200/) (2020)
- [Neural Autoregressive Flows](https://mlsanity.com/paper/1804.00779/) (2018)
- [Unsupervised Degradation Representation Learning for Blind Super-Resolution](https://mlsanity.com/paper/2104.00416/) (2021)
- [Learning Likelihoods with Conditional Normalizing Flows](https://mlsanity.com/paper/1912.00042/) (2019)
- [Guided Image Generation with Conditional Invertible Neural Networks](https://mlsanity.com/paper/1907.02392/) (2019)
- [RankSRGAN: Generative Adversarial Networks with Ranker for Image Super-Resolution](https://mlsanity.com/paper/1908.06382/) (2019)
- [Video Super-Resolution Transformer](https://mlsanity.com/paper/2106.06847/) (2021)
- [Video Super-Resolution with Recurrent Structure-Detail Network](https://mlsanity.com/paper/2008.00455/) (2020)
- [Invertible Image Rescaling](https://mlsanity.com/paper/2005.05650/) (2020)
- [Unsupervised Real-world Image Super Resolution via Domain-distance Aware Training](https://mlsanity.com/paper/2004.01178/) (2020)

## Built on by (most cited first, 30 of 73)

- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [VRT: A Video Restoration Transformer](https://mlsanity.com/paper/2201.12288/) (2022)
- [Implicit Diffusion Models for Continuous Super-Resolution](https://mlsanity.com/paper/2303.16491/) (2023)
- [Conditional Image Generation with Score-Based Diffusion Models](https://mlsanity.com/paper/2111.13606/) (2021)
- [Recurrent Video Restoration Transformer with Guided Deformable Attention](https://mlsanity.com/paper/2206.02146/) (2022)
- [Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution](https://mlsanity.com/paper/2203.09195/) (2022)
- [CiaoSR: Continuous Implicit Attention-in-Attention Network for Arbitrary-Scale Image Super-Resolution](https://mlsanity.com/paper/2212.04362/) (2022)
- [ResDiff: Combining CNN and Diffusion Model for Image Super-Resolution](https://mlsanity.com/paper/2303.08714/) (2023)
- [Reference-based Image Super-Resolution with Deformable Attention Transformer](https://mlsanity.com/paper/2207.11938/) (2022)
- [Local Implicit Normalizing Flow for Arbitrary-Scale Image Super-Resolution](https://mlsanity.com/paper/2303.05156/) (2023)
- [PatchNR: Learning from Very Few Images by Patch Normalizing Flow Regularization](https://mlsanity.com/paper/2205.12021/) (2022)
- [DiSR-NeRF: Diffusion-Guided View-Consistent Super-Resolution NeRF](https://mlsanity.com/paper/2404.00874/) (2024)
- [CDPMSR: Conditional Diffusion Probabilistic Models for Single Image Super-Resolution](https://mlsanity.com/paper/2302.12831/) (2023)
- [Low-field magnetic resonance image enhancement via stochastic image quality transfer](https://mlsanity.com/paper/2304.13385/) (2023)
- [Target-aware Molecular Graph Generation](https://mlsanity.com/paper/2202.04829/) (2022)
- [ACDMSR: Accelerated Conditional Diffusion Models for Single Image Super-Resolution](https://mlsanity.com/paper/2307.00781/) (2023)
- [Binarized Diffusion Model for Image Super-Resolution](https://mlsanity.com/paper/2406.05723/) (2024)
- [Non-Uniform Diffusion Models](https://mlsanity.com/paper/2207.09786/) (2022)
- [DREAM: Diffusion Rectification and Estimation-Adaptive Models](https://mlsanity.com/paper/2312.00210/) (2023)
- [Semantic Guided Large Scale Factor Remote Sensing Image Super-resolution with Generative Diffusion Prior](https://mlsanity.com/paper/2405.07044/) (2024)
- [Normalizing Flow as a Flexible Fidelity Objective for Photo-Realistic Super-resolution](https://mlsanity.com/paper/2111.03649/) (2021)
- [Dynamic Attention-Guided Diffusion for Image Super-Resolution](https://mlsanity.com/paper/2308.07977/) (2023)
- [SAM-DiffSR: Structure-Modulated Diffusion Model for Image Super-Resolution](https://mlsanity.com/paper/2402.17133/) (2024)
- [Self-Asymmetric Invertible Network for Compression-Aware Image Rescaling](https://mlsanity.com/paper/2303.02353/) (2023)
- [A Dive into SAM Prior in Image Restoration](https://mlsanity.com/paper/2305.13620/) (2023)
- [FS-NCSR: Increasing Diversity of the Super-Resolution Space via Frequency Separation and Noise-Conditioned Normalizing Flow](https://mlsanity.com/paper/2204.09679/) (2022)
- [Downscaled Representation Matters: Improving Image Rescaling with Collaborative Downscaled Images](https://mlsanity.com/paper/2211.10643/) (2022)
- [On the Robustness of Normalizing Flows for Inverse Problems in Imaging](https://mlsanity.com/paper/2212.04319/) (2022)
- [PSRFlow: Probabilistic Super Resolution with Flow-Based Models for Scientific Data](https://mlsanity.com/paper/2308.04605/) (2023)
- [HoliSDiP: Image Super-Resolution via Holistic Semantics and Diffusion Prior](https://mlsanity.com/paper/2411.18662/) (2024)

## Abstract (arXiv)

Normalizing flows have recently demonstrated promising results for low-level vision tasks. For image super-resolution (SR), it learns to predict diverse photo-realistic high-resolution (HR) images from the low-resolution (LR) image rather than learning a deterministic mapping. For image rescaling, it achieves high accuracy by jointly modelling the downscaling and upscaling processes. While existing approaches employ specialized techniques for these two tasks, we set out to unify them in a single formulation. In this paper, we propose the hierarchical conditional flow (HCFlow) as a unified framework for image SR and image rescaling. More specifically, HCFlow learns a bijective mapping between HR and LR image pairs by modelling the distribution of the LR image and the rest high-frequency component simultaneously. In particular, the high-frequency component is conditional on the LR image in a hierarchical manner. To further enhance the performance, other losses such as perceptual loss and GAN loss are combined with the commonly used negative log-likelihood loss in training. Extensive experiments on general image SR, face image SR and image rescaling have demonstrated that the proposed HCFlow achieves state-of-the-art performance in terms of both quantitative metrics and visual quality.