# Deep Fourier Up-Sampling

arXiv: https://arxiv.org/abs/2210.05171 · HTML: https://mlsanity.com/paper/2210.05171/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 55 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [Benchmarking Single Image Dehazing and Beyond](https://mlsanity.com/paper/1712.04143/) (2017)
- [Progressive Image Deraining Networks: A Better and Simpler Baseline](https://mlsanity.com/paper/1901.09221/) (2019)
- [Density-aware Single Image De-raining using a Multi-stream Dense Network](https://mlsanity.com/paper/1802.07412/) (2018)
- [Multi-Scale Boosted Dehazing Network with Dense Feature Fusion](https://mlsanity.com/paper/2004.13388/) (2020)
- [Image De-raining Using a Conditional Generative Adversarial Network](https://mlsanity.com/paper/1701.05957/) (2017)
- [Variational Denoising Network: Toward Blind Noise Modeling and Removal](https://mlsanity.com/paper/1908.11314/) (2019)
- [Spectral Representations for Convolutional Neural Networks](https://mlsanity.com/paper/1506.03767/) (2015)
- [Lightweight Pyramid Networks for Image Deraining](https://mlsanity.com/paper/1805.06173/) (2018)
- [CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices](https://mlsanity.com/paper/1708.08917/) (2017)
- [Domain Adaptive Relational Reasoning for 3D Multi-Organ Segmentation](https://mlsanity.com/paper/2005.09120/) (2020)
- [Learning Dual Convolutional Neural Networks for Low-Level Vision](https://mlsanity.com/paper/1805.05020/) (2018)
- [Memory-augmented Deep Unfolding Network for Guided Image Super-resolution](https://mlsanity.com/paper/2203.04960/) (2022)
- [Feature Pyramid Network for Multi-Class Land Segmentation](https://mlsanity.com/paper/1806.03510/) (2018)
- [Neural Sparse Representation for Image Restoration](https://mlsanity.com/paper/2006.04357/) (2020)
- [FALCON: A Fourier Transform Based Approach for Fast and Secure Convolutional Neural Network Predictions](https://mlsanity.com/paper/1811.08257/) (2018)
- [IPG-Net: Image Pyramid Guidance Network for Small Object Detection](https://mlsanity.com/paper/1912.00632/) (2019)
- [Unfolding Taylor's Approximations for Image Restoration](https://mlsanity.com/paper/2109.03442/) (2021)

## Built on by (most cited first, 27 of 27)

- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [FourLLIE: Boosting Low-Light Image Enhancement by Fourier Frequency Information](https://mlsanity.com/paper/2308.03033/) (2023)
- [Sparse Sampling Transformer with Uncertainty-Driven Ranking for Unified Removal of Raindrops and Rain Streaks](https://mlsanity.com/paper/2308.14153/) (2023)
- [NightHazeFormer: Single Nighttime Haze Removal Using Prior Query Transformer](https://mlsanity.com/paper/2305.09533/) (2023)
- [3D-GSW: 3D Gaussian Splatting for Robust Watermarking](https://mlsanity.com/paper/2409.13222/) (2024)
- [OPE-SR: Orthogonal Position Encoding for Designing a Parameter-free Upsampling Module in Arbitrary-scale Image Super-Resolution](https://mlsanity.com/paper/2303.01091/) (2023)
- [FreMIM: Fourier Transform Meets Masked Image Modeling for Medical Image Segmentation](https://mlsanity.com/paper/2304.10864/) (2023)
- [Hierarchical Neural Operator Transformer with Learnable Frequency-aware Loss Prior for Arbitrary-scale Super-resolution](https://mlsanity.com/paper/2405.12202/) (2024)
- [Hybrid Transformer and CNN Attention Network for Stereo Image Super-resolution](https://mlsanity.com/paper/2305.05177/) (2023)
- [PPTFormer: Pseudo Multi-Perspective Transformer for UAV Segmentation](https://mlsanity.com/paper/2406.19632/) (2024)
- [OPDN: Omnidirectional Position-aware Deformable Network for Omnidirectional Image Super-Resolution](https://mlsanity.com/paper/2304.13471/) (2023)
- [EmMixformer: Mix transformer for eye movement recognition](https://mlsanity.com/paper/2401.04956/) (2024)
- [Resolution-Invariant Image Classification based on Fourier Neural Operators](https://mlsanity.com/paper/2304.01227/) (2023)
- [Noise Matters: Optimizing Matching Noise for Diffusion Classifiers](https://mlsanity.com/paper/2508.11330/) (2025)
- [Local-Global Transformer Enhanced Unfolding Network for Pan-sharpening](https://mlsanity.com/paper/2304.14612/) (2023)
- [RSHazeDiff: A Unified Fourier-aware Diffusion Model for Remote Sensing Image Dehazing](https://mlsanity.com/paper/2405.09083/) (2024)
- [Learned Image Reasoning Prior Penetrates Deep Unfolding Network for Panchromatic and Multi-Spectral Image Fusion](https://mlsanity.com/paper/2308.16083/) (2023)
- [Pixel Adapter: A Graph-Based Post-Processing Approach for Scene Text Image Super-Resolution](https://mlsanity.com/paper/2309.08919/) (2023)
- [I3Net: Inter-Intra-slice Interpolation Network for Medical Slice Synthesis](https://mlsanity.com/paper/2405.02857/) (2024)
- [Decomposition Ascribed Synergistic Learning for Unified Image Restoration](https://mlsanity.com/paper/2308.00759/) (2023)
- [MacFormer: Semantic Segmentation with Fine Object Boundaries](https://mlsanity.com/paper/2408.05699/) (2024)
- [A Dual Domain Multi-exposure Image Fusion Network based on the Spatial-Frequency Integration](https://mlsanity.com/paper/2312.10604/) (2023)
- [Enhancing Infrared Small Target Detection Robustness with Bi-Level Adversarial Framework](https://mlsanity.com/paper/2309.01099/) (2023)
- [UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representations](https://mlsanity.com/paper/2510.24116/) (2025)
- [Textless NLP -- Zero Resource Challenge with Low Resource Compute](https://mlsanity.com/paper/2409.19015/) (2024)
- [DeepRFTv2: Kernel-level Learning for Image Deblurring](https://mlsanity.com/paper/2511.21132/) (2025)
- [FSM-Net: An Efficient Frequency-Spatial Network for Real-World Deblurring](https://mlsanity.com/paper/2605.31400/) (2026)

## Abstract (arXiv)

Existing convolutional neural networks widely adopt spatial down-/up-sampling for multi-scale modeling. However, spatial up-sampling operators (e.g., interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, incapably exploring the global dependency. In contrast, the Fourier domain obeys the nature of global modeling according to the spectral convolution theorem. Unlike the spatial domain that performs up-sampling with the property of local similarity, up-sampling in the Fourier domain is more challenging as it does not follow such a local property. In this study, we propose a theoretically sound Deep Fourier Up-Sampling (FourierUp) to solve these issues. We revisit the relationships between spatial and Fourier domains and reveal the transform rules on the features of different resolutions in the Fourier domain, which provide key insights for FourierUp's designs. FourierUp as a generic operator consists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform, which can be directly integrated with existing networks. Extensive experiments across multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution, demonstrate the consistent performance gains obtained by introducing our FourierUp.