# Semantic Layout Manipulation with High-Resolution Sparse Attention

arXiv: https://arxiv.org/abs/2012.07288 · HTML: https://mlsanity.com/paper/2012.07288/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 10 of 10)

- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Compositional Transformers for Scene Generation](https://mlsanity.com/paper/2111.08960/) (2021)
- [Embedding Arithmetic of Multimodal Queries for Image Retrieval](https://mlsanity.com/paper/2112.03162/) (2021)
- [Bi-level Feature Alignment for Versatile Image Translation and Manipulation](https://mlsanity.com/paper/2107.03021/) (2021)
- [Marginal Contrastive Correspondence for Guided Image Generation](https://mlsanity.com/paper/2204.00442/) (2022)
- [Eliminating Contextual Prior Bias for Semantic Image Editing via Dual-Cycle Diffusion](https://mlsanity.com/paper/2302.02394/) (2023)
- [DragGaussian: Enabling Drag-style Manipulation on 3D Gaussian Representation](https://mlsanity.com/paper/2405.05800/) (2024)
- [ASSET: Autoregressive Semantic Scene Editing with Transformers at High Resolutions](https://mlsanity.com/paper/2205.12231/) (2022)
- [Structure-Guided Image Completion with Image-level and Object-level Semantic Discriminators](https://mlsanity.com/paper/2212.06310/) (2022)
- [Domain-Scalable Unpaired Image Translation via Latent Space Anchoring](https://mlsanity.com/paper/2306.14879/) (2023)

## Abstract (arXiv)

We tackle the problem of semantic image layout manipulation, which aims to manipulate an input image by editing its semantic label map. A core problem of this task is how to transfer visual details from the input images to the new semantic layout while making the resulting image visually realistic. Recent work on learning cross-domain correspondence has shown promising results for global layout transfer with dense attention-based warping. However, this method tends to lose texture details due to the resolution limitation and the lack of smoothness constraint of correspondence. To adapt this paradigm for the layout manipulation task, we propose a high-resolution sparse attention module that effectively transfers visual details to new layouts at a resolution up to 512x512. To further improve visual quality, we introduce a novel generator architecture consisting of a semantic encoder and a two-stage decoder for coarse-to-fine synthesis. Experiments on the ADE20k and Places365 datasets demonstrate that our proposed approach achieves substantial improvements over the existing inpainting and layout manipulation methods.