# In&Out : Diverse Image Outpainting via GAN Inversion

arXiv: https://arxiv.org/abs/2104.00675 · HTML: https://mlsanity.com/paper/2104.00675/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 48 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 30 of 38)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [In-Domain GAN Inversion for Real Image Editing](https://mlsanity.com/paper/2004.00049/) (2020)
- [Image2StyleGAN++: How to Edit the Embedded Images?](https://mlsanity.com/paper/1911.11544/) (2019)
- [Adversarial Feature Learning](https://mlsanity.com/paper/1605.09782/) (2016)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1808.00948/) (2018)
- [Adversarially Learned Inference](https://mlsanity.com/paper/1606.00704/) (2016)
- [Semantic Photo Manipulation with a Generative Image Prior](https://mlsanity.com/paper/2005.07727/) (2020)
- [Video Generation From Text](https://mlsanity.com/paper/1710.00421/) (2017)
- [Inverting The Generator Of A Generative Adversarial Network](https://mlsanity.com/paper/1611.05644/) (2016)
- [Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis](https://mlsanity.com/paper/1903.05628/) (2019)
- [Invertible Conditional GANs for image editing](https://mlsanity.com/paper/1611.06355/) (2016)
- [Neural Photo Editing with Introspective Adversarial Networks](https://mlsanity.com/paper/1609.07093/) (2016)

## Built on by (most cited first, 30 of 55)

- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting](https://mlsanity.com/paper/2312.03594/) (2023)
- [Regularizing Generative Adversarial Networks under Limited Data](https://mlsanity.com/paper/2104.03310/) (2021)
- [InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images](https://mlsanity.com/paper/2207.11148/) (2022)
- [AVID: Any-Length Video Inpainting with Diffusion Model](https://mlsanity.com/paper/2312.03816/) (2023)
- [Any-resolution Training for High-resolution Image Synthesis](https://mlsanity.com/paper/2204.07156/) (2022)
- [InfinityGAN: Towards Infinite-Pixel Image Synthesis](https://mlsanity.com/paper/2104.03963/) (2021)
- [360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model](https://mlsanity.com/paper/2401.06578/) (2024)
- [SemCity: Semantic Scene Generation with Triplane Diffusion](https://mlsanity.com/paper/2403.07773/) (2024)
- [Customizing 360-Degree Panoramas through Text-to-Image Diffusion Models](https://mlsanity.com/paper/2310.18840/) (2023)
- [Persistent Nature: A Generative Model of Unbounded 3D Worlds](https://mlsanity.com/paper/2303.13515/) (2023)
- [Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive Content Generation](https://mlsanity.com/paper/2409.01055/) (2024)
- [3DAvatarGAN: Bridging Domains for Personalized Editable Avatars](https://mlsanity.com/paper/2301.02700/) (2023)
- [Amodal Completion via Progressive Mixed Context Diffusion](https://mlsanity.com/paper/2312.15540/) (2023)
- [Guided Co-Modulated GAN for 360deg Field of View Extrapolation](https://mlsanity.com/paper/2204.07286/) (2022)
- [DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathology](https://mlsanity.com/paper/2306.13384/) (2023)
- [Imagine360: Immersive 360 Video Generation from Perspective Anchor](https://mlsanity.com/paper/2412.03552/) (2024)
- [Hierarchical Masked 3D Diffusion Model for Video Outpainting](https://mlsanity.com/paper/2309.02119/) (2023)
- [Arbitrary-Scale Image Synthesis](https://mlsanity.com/paper/2204.02273/) (2022)
- [Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach](https://mlsanity.com/paper/2401.15652/) (2024)
- [PanoFree: Tuning-Free Holistic Multi-view Image Generation with Cross-view Self-Guidance](https://mlsanity.com/paper/2408.02157/) (2024)
- [Vector Quantized Image-to-Image Translation](https://mlsanity.com/paper/2207.13286/) (2022)
- [High-Fidelity Image Inpainting with GAN Inversion](https://mlsanity.com/paper/2208.11850/) (2022)
- [StyleGAN of All Trades: Image Manipulation with Only Pretrained StyleGAN](https://mlsanity.com/paper/2111.01619/) (2021)
- [PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Era](https://mlsanity.com/paper/2509.12989/) (2025)
- [LFS-GAN: Lifelong Few-Shot Image Generation](https://mlsanity.com/paper/2308.11917/) (2023)
- [CubeComposer: Spatio-Temporal Autoregressive 4K 360deg Video Generation from Perspective Video](https://mlsanity.com/paper/2603.04291/) (2026)
- [Conditional 360-degree Image Synthesis for Immersive Indoor Scene Decoration](https://mlsanity.com/paper/2307.09621/) (2023)
- [Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360 Image Outpainting](https://mlsanity.com/paper/2401.10564/) (2024)

## Abstract (arXiv)

Image outpainting seeks for a semantically consistent extension of the input image beyond its available content. Compared to inpainting -- filling in missing pixels in a way coherent with the neighboring pixels -- outpainting can be achieved in more diverse ways since the problem is less constrained by the surrounding pixels. Existing image outpainting methods pose the problem as a conditional image-to-image translation task, often generating repetitive structures and textures by replicating the content available in the input image. In this work, we formulate the problem from the perspective of inverting generative adversarial networks. Our generator renders micro-patches conditioned on their joint latent code as well as their individual positions in the image. To outpaint an image, we seek for multiple latent codes not only recovering available patches but also synthesizing diverse outpainting by patch-based generation. This leads to richer structure and content in the outpainted regions. Furthermore, our formulation allows for outpainting conditioned on the categorical input, thereby enabling flexible user controls. Extensive experimental results demonstrate the proposed method performs favorably against existing in- and outpainting methods, featuring higher visual quality and diversity.