# Downscaled Representation Matters: Improving Image Rescaling with Collaborative Downscaled Images

arXiv: https://arxiv.org/abs/2211.10643 · HTML: https://mlsanity.com/paper/2211.10643/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 75 references that are in the map.
- First posted on arXiv 19 Nov 2022.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Sketch-based Manga Retrieval using Manga109 Dataset](https://mlsanity.com/paper/1510.04389/) (2015)
- [Learning a Single Convolutional Super-Resolution Network for Multiple Degradations](https://mlsanity.com/paper/1712.06116/) (2017)
- [Discrimination-aware Channel Pruning for Deep Neural Networks](https://mlsanity.com/paper/1810.11809/) (2018)
- [Understanding Convolution for Semantic Segmentation](https://mlsanity.com/paper/1702.08502/) (2017)
- [Feedback Network for Image Super-Resolution](https://mlsanity.com/paper/1903.09814/) (2019)
- [Transformer for Single Image Super-Resolution](https://mlsanity.com/paper/2108.11084/) (2021)
- [Local Texture Estimator for Implicit Representation Function](https://mlsanity.com/paper/2111.08918/) (2021)
- [Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernels](https://mlsanity.com/paper/1903.12529/) (2019)
- [Invertible Image Rescaling](https://mlsanity.com/paper/2005.05650/) (2020)
- [Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolution](https://mlsanity.com/paper/2003.07018/) (2020)
- [Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling](https://mlsanity.com/paper/2108.05301/) (2021)
- [Real-World Single Image Super-Resolution: A Brief Review](https://mlsanity.com/paper/2103.02368/) (2021)
- [Learned Image Downscaling for Upscaling using Content Adaptive Resampler](https://mlsanity.com/paper/1907.12904/) (2019)
- [Discrimination-aware Network Pruning for Deep Model Compression](https://mlsanity.com/paper/2001.01050/) (2020)
- [Contrastive Neural Architecture Search with Neural Architecture Comparators](https://mlsanity.com/paper/2103.05471/) (2021)
- [Simultaneous Super-Resolution and Cross-Modality Synthesis of 3D Medical Images using Weakly-Supervised Joint Convolutional Sparse Coding](https://mlsanity.com/paper/1705.02596/) (2017)
- [NAT: Neural Architecture Transformer for Accurate and Compact Architectures](https://mlsanity.com/paper/1910.14488/) (2019)

## Built on by (most cited first, 8 of 8)

- [Robustifying Token Attention for Vision Transformers](https://mlsanity.com/paper/2303.11126/) (2023)
- [Learning Many-to-Many Mapping for Unpaired Real-World Image Super-resolution and Downscaling](https://mlsanity.com/paper/2310.04964/) (2023)
- [ResVR: Joint Rescaling and Viewport Rendering of Omnidirectional Images](https://mlsanity.com/paper/2404.16825/) (2024)
- [Boosting Semi-Supervised Learning with Contrastive Complementary Labeling](https://mlsanity.com/paper/2212.06643/) (2022)
- [Timestep-Aware Diffusion Model for Extreme Image Rescaling](https://mlsanity.com/paper/2408.09151/) (2024)
- [Test-time adaptation for image compression with distribution regularization](https://mlsanity.com/paper/2410.12191/) (2024)
- [Learning Arbitrary-Scale RAW Image Downscaling with Wavelet-based Recurrent Reconstruction](https://mlsanity.com/paper/2507.23219/) (2025)
- [Block-Based Multi-Scale Image Rescaling](https://mlsanity.com/paper/2412.11468/) (2024)

## Abstract (arXiv)

Deep networks have achieved great success in image rescaling (IR) task that seeks to learn the optimal downscaled representations, i.e., low-resolution (LR) images, to reconstruct the original high-resolution (HR) images. Compared with super-resolution methods that consider a fixed downscaling scheme, e.g., bicubic, IR often achieves significantly better reconstruction performance thanks to the learned downscaled representations. This highlights the importance of a good downscaled representation in image reconstruction tasks. Existing IR methods mainly learn the downscaled representation by jointly optimizing the downscaling and upscaling models. Unlike them, we seek to improve the downscaled representation through a different and more direct way: optimizing the downscaled image itself instead of the down-/upscaling models. Specifically, we propose a collaborative downscaling scheme that directly generates the collaborative LR examples by descending the gradient w.r.t. the reconstruction loss on them to benefit the IR process. Furthermore, since LR images are downscaled from the corresponding HR images, one can also improve the downscaled representation if we have a better representation in the HR domain. Inspired by this, we propose a Hierarchical Collaborative Downscaling (HCD) method that performs gradient descent in both HR and LR domains to improve the downscaled representations. Extensive experiments show that our HCD significantly improves the reconstruction performance both quantitatively and qualitatively. Moreover, we also highlight the flexibility of our HCD since it can generalize well across diverse IR models.