# User-Controllable Latent Transformer for StyleGAN Image Layout Editing

arXiv: https://arxiv.org/abs/2208.12408 · HTML: https://mlsanity.com/paper/2208.12408/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 43 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows](https://mlsanity.com/paper/2008.02401/) (2020)
- [StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation](https://mlsanity.com/paper/2011.12799/) (2020)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [StyleRig: Rigging StyleGAN for 3D Control over Portrait Images](https://mlsanity.com/paper/2004.00121/) (2020)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [On the "steerability" of generative adversarial networks](https://mlsanity.com/paper/1907.07171/) (2019)
- [ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement](https://mlsanity.com/paper/2104.02699/) (2021)
- [Editing in Style: Uncovering the Local Semantics of GANs](https://mlsanity.com/paper/2004.14367/) (2020)
- [Repurposing GANs for One-shot Semantic Part Segmentation](https://mlsanity.com/paper/2103.04379/) (2021)
- [A Latent Transformer for Disentangled Face Editing in Images and Videos](https://mlsanity.com/paper/2106.11895/) (2021)

## Built on by (most cited first, 28 of 28)

- [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://mlsanity.com/paper/2306.14435/) (2023)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [Wear-Any-Way: Manipulable Virtual Try-on via Sparse Correspondence Alignment](https://mlsanity.com/paper/2403.12965/) (2024)
- [PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions](https://mlsanity.com/paper/2308.05184/) (2023)
- [RegionDrag: Fast Region-Based Image Editing with Diffusion Models](https://mlsanity.com/paper/2407.18247/) (2024)
- [FreeDrag: Feature Dragging for Reliable Point-based Image Editing](https://mlsanity.com/paper/2307.04684/) (2023)
- [StableDrag: Stable Dragging for Point-based Image Editing](https://mlsanity.com/paper/2403.04437/) (2024)
- [Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models](https://mlsanity.com/paper/2310.07653/) (2023)
- [DiffUHaul: A Training-Free Method for Object Dragging in Images](https://mlsanity.com/paper/2406.01594/) (2024)
- [LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videos](https://mlsanity.com/paper/2405.13722/) (2024)
- [IntrinsicEdit: Precise generative image manipulation in intrinsic space](https://mlsanity.com/paper/2505.08889/) (2025)
- [A Survey on Deep Generative 3D-aware Image Synthesis](https://mlsanity.com/paper/2210.14267/) (2022)
- [DragText: Rethinking Text Embedding in Point-based Image Editing](https://mlsanity.com/paper/2407.17843/) (2024)
- [Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generation](https://mlsanity.com/paper/2308.06027/) (2023)
- [AdaptiveDrag: Semantic-Driven Dragging on Diffusion-Based Image Editing](https://mlsanity.com/paper/2410.12696/) (2024)
- [DragGaussian: Enabling Drag-style Manipulation on 3D Gaussian Representation](https://mlsanity.com/paper/2405.05800/) (2024)
- [Inpaint4Drag: Repurposing Inpainting Models for Drag-Based Image Editing via Bidirectional Warping](https://mlsanity.com/paper/2509.04582/) (2025)
- [Edit One for All: Interactive Batch Image Editing](https://mlsanity.com/paper/2401.10219/) (2024)
- [PixelMan: Consistent Object Editing with Diffusion Models via Pixel Manipulation and Generation](https://mlsanity.com/paper/2412.14283/) (2024)
- [Blocks2World: Controlling Realistic Scenes with Editable Primitives](https://mlsanity.com/paper/2307.03847/) (2023)
- [3DGS-Drag: Dragging Gaussians for Intuitive Point-Based 3D Editing](https://mlsanity.com/paper/2601.07963/) (2026)
- [Identity-preserving Editing of Multiple Facial Attributes by Learning Global Edit Directions and Local Adjustments](https://mlsanity.com/paper/2309.14267/) (2023)
- [Auto DragGAN: Editing the Generative Image Manifold in an Autoregressive Manner](https://mlsanity.com/paper/2407.18656/) (2024)
- [Revealing Directions for Text-guided 3D Face Editing](https://mlsanity.com/paper/2410.04965/) (2024)
- [Exploring Attribute Variations in Style-based GANs using Diffusion Models](https://mlsanity.com/paper/2311.16052/) (2023)
- [Key-point Guided Deformable Image Manipulation Using Diffusion Model](https://mlsanity.com/paper/2401.08178/) (2024)
- [DynaDrag: Dynamic Drag-Style Image Editing by Motion Prediction](https://mlsanity.com/paper/2601.00542/) (2026)
- [ToW3D: Consistency-aware Interactive Point-based Mesh Editing on GANs](https://mlsanity.com/paper/2609.26078/) (2026)

## Abstract (arXiv)

Latent space exploration is a technique that discovers interpretable latent directions and manipulates latent codes to edit various attributes in images generated by generative adversarial networks (GANs). However, in previous work, spatial control is limited to simple transformations (e.g., translation and rotation), and it is laborious to identify appropriate latent directions and adjust their parameters. In this paper, we tackle the problem of editing the StyleGAN image layout by annotating the image directly. To do so, we propose an interactive framework for manipulating latent codes in accordance with the user inputs. In our framework, the user annotates a StyleGAN image with locations they want to move or not and specifies a movement direction by mouse dragging. From these user inputs and initial latent codes, our latent transformer based on a transformer encoder-decoder architecture estimates the output latent codes, which are fed to the StyleGAN generator to obtain a result image. To train our latent transformer, we utilize synthetic data and pseudo-user inputs generated by off-the-shelf StyleGAN and optical flow models, without manual supervision. Quantitative and qualitative evaluations demonstrate the effectiveness of our method over existing methods.