# Both Spatial and Frequency Cues Contribute to High-Fidelity Image Inpainting

arXiv: https://arxiv.org/abs/2307.07678 · HTML: https://mlsanity.com/paper/2307.07678/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 15 Jul 2023.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions](https://mlsanity.com/paper/2003.01826/) (2020)
- [MAT: Mask-Aware Transformer for Large Hole Image Inpainting](https://mlsanity.com/paper/2203.15270/) (2022)
- [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020)
- [Learning in the Frequency Domain](https://mlsanity.com/paper/2002.12416/) (2020)
- [Training behavior of deep neural network in frequency domain](https://mlsanity.com/paper/1807.01251/) (2018)
- [Image Inpainting via Generative Multi-column Convolutional Neural Networks](https://mlsanity.com/paper/1810.08771/) (2018)
- [On the Frequency Bias of Generative Models](https://mlsanity.com/paper/2111.02447/) (2021)
- [WaveFill: A Wavelet-based Generation Network for Image Inpainting](https://mlsanity.com/paper/2107.11027/) (2021)
- [Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translation](https://mlsanity.com/paper/2203.06321/) (2022)
- [High-Fidelity and Arbitrary Face Editing](https://mlsanity.com/paper/2103.15814/) (2021)
- [Image Inpainting by End-to-End Cascaded Refinement with Mask Awareness](https://mlsanity.com/paper/2104.13743/) (2021)
- [Wavelet Domain Style Transfer for an Effective Perception-distortion Tradeoff in Single Image Super-Resolution](https://mlsanity.com/paper/1910.04074/) (2019)
- [Multi-level Wavelet-based Generative Adversarial Network for Perceptual Quality Enhancement of Compressed Video](https://mlsanity.com/paper/2008.00499/) (2020)
- [Image Fine-grained Inpainting](https://mlsanity.com/paper/2002.02609/) (2020)
- [Deep Generative Model for Image Inpainting with Local Binary Pattern Learning and Spatial Attention](https://mlsanity.com/paper/2009.01031/) (2020)
- [GLaMa: Joint Spatial and Frequency Loss for General Image Inpainting](https://mlsanity.com/paper/2205.07162/) (2022)
- [Zoom-to-Inpaint: Image Inpainting with High-Frequency Details](https://mlsanity.com/paper/2012.09401/) (2020)

## Abstract (arXiv)

Deep generative approaches have obtained great success in image inpainting recently. However, most generative inpainting networks suffer from either over-smooth results or aliasing artifacts. The former lacks high-frequency details, while the latter lacks semantic structure. To address this issue, we propose an effective Frequency-Spatial Complementary Network (FSCN) by exploiting rich semantic information in both spatial and frequency domains. Specifically, we introduce an extra Frequency Branch and Frequency Loss on the spatial-based network to impose direct supervision on the frequency information, and propose a Frequency-Spatial Cross-Attention Block (FSCAB) to fuse multi-domain features and combine the corresponding characteristics. With our FSCAB, the inpainting network is capable of capturing frequency information and preserving visual consistency simultaneously. Extensive quantitative and qualitative experiments demonstrate that our inpainting network can effectively achieve superior results, outperforming previous state-of-the-art approaches with significantly fewer parameters and less computation cost. The code will be released soon.