# Learning to Manipulate Individual Objects in an Image

arXiv: https://arxiv.org/abs/2004.05495 · HTML: https://mlsanity.com/paper/2004.05495/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2020.

## Built on by (most cited first, 22 of 22)

- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)
- [ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/2111.10265/) (2021)
- [Generalization and Robustness Implications in Object-Centric Learning](https://mlsanity.com/paper/2107.00637/) (2021)
- [SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editing](https://mlsanity.com/paper/2112.02236/) (2021)
- [Unsupervised Layered Image Decomposition into Object Prototypes](https://mlsanity.com/paper/2104.14575/) (2021)
- [Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representations](https://mlsanity.com/paper/2106.03630/) (2021)
- [Compositional Scene Representation Learning via Reconstruction: A Survey](https://mlsanity.com/paper/2202.07135/) (2022)
- [Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learning](https://mlsanity.com/paper/2301.05169/) (2023)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [Inductive Biases for Object-Centric Representations in the Presence of Complex Textures](https://mlsanity.com/paper/2204.08479/) (2022)
- [Semantic 3D-aware Portrait Synthesis and Manipulation Based on Compositional Neural Radiance Field](https://mlsanity.com/paper/2302.01579/) (2023)
- [BlobGAN-3D: A Spatially-Disentangled 3D-Aware Generative Model for Indoor Scenes](https://mlsanity.com/paper/2303.14706/) (2023)
- [Knowledge-Guided Object Discovery with Acquired Deep Impressions](https://mlsanity.com/paper/2103.10611/) (2021)
- [Unsupervised Object Discovery: A Comprehensive Survey and Unified Taxonomy](https://mlsanity.com/paper/2411.00868/) (2024)
- [The Learnable Typewriter: A Generative Approach to Text Analysis](https://mlsanity.com/paper/2302.01660/) (2023)
- [Towards Learning Geometric Eigen-Lengths Crucial for Fitting Tasks](https://mlsanity.com/paper/2312.15610/) (2023)
- [MetaPix: Domain Transfer for Semantic Segmentation by Meta Pixel Weighting](https://mlsanity.com/paper/2110.01777/) (2021)
- [Object Pursuit: Building a Space of Objects via Discriminative Weight Generation](https://mlsanity.com/paper/2112.07954/) (2021)
- [Compositional Scene Modeling with Global Object-Centric Representations](https://mlsanity.com/paper/2211.11500/) (2022)
- [Multiple Invertible and Partial-Equivariant Function for Latent Vector Transformation to Enhance Disentanglement in VAEs](https://mlsanity.com/paper/2502.03740/) (2025)

## Abstract (arXiv)

We describe a method to train a generative model with latent factors that are (approximately) independent and localized. This means that perturbing the latent variables affects only local regions of the synthesized image, corresponding to objects. Unlike other unsupervised generative models, ours enables object-centric manipulation, without requiring object-level annotations, or any form of annotation for that matter. The key to our method is the combination of spatial disentanglement, enforced by a Contextual Information Separation loss, and perceptual cycle-consistency, enforced by a loss that penalizes changes in the image partition in response to perturbations of the latent factors. We test our method's ability to allow independent control of spatial and semantic factors of variability on existing datasets and also introduce two new ones that highlight the limitations of current methods.