# Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representations

arXiv: https://arxiv.org/abs/2106.03630 · HTML: https://mlsanity.com/paper/2106.03630/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Neural Expectation Maximization](https://mlsanity.com/paper/1708.03498/) (2017)
- [Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs](https://mlsanity.com/paper/1901.07017/) (2019)
- [Entity Abstraction in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/1910.12827/) (2019)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)
- [Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions](https://mlsanity.com/paper/1802.10353/) (2018)
- [Iterative Amortized Inference](https://mlsanity.com/paper/1807.09356/) (2018)
- [The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement](https://mlsanity.com/paper/2008.10599/) (2020)
- [Taming VAEs](https://mlsanity.com/paper/1810.00597/) (2018)
- [COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration](https://mlsanity.com/paper/1905.09275/) (2019)

## Built on by (most cited first, 30 of 36)

- [Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization](https://mlsanity.com/paper/2205.07839/) (2022)
- [ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/2111.10265/) (2021)
- [Unsupervised Part Discovery from Contrastive Reconstruction](https://mlsanity.com/paper/2111.06349/) (2021)
- [Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames](https://mlsanity.com/paper/2302.04973/) (2023)
- [Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion](https://mlsanity.com/paper/2205.07844/) (2022)
- [Compositional Scene Representation Learning via Reconstruction: A Survey](https://mlsanity.com/paper/2202.07135/) (2022)
- [Slot-VAE: Object-Centric Scene Generation with Slot Attention](https://mlsanity.com/paper/2306.06997/) (2023)
- [Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos](https://mlsanity.com/paper/2308.09951/) (2023)
- [Unsupervised Multi-object Segmentation by Predicting Probable Motion Patterns](https://mlsanity.com/paper/2210.12148/) (2022)
- [Contrastive Training of Complex-Valued Autoencoders for Object Discovery](https://mlsanity.com/paper/2305.15001/) (2023)
- [Multi-Object Discovery by Low-Dimensional Object Motion](https://mlsanity.com/paper/2307.08027/) (2023)
- [Motion-inductive Self-supervised Object Discovery in Videos](https://mlsanity.com/paper/2210.00221/) (2022)
- [Unsupervised Learning of Compositional Scene Representations from Multiple Unspecified Viewpoints](https://mlsanity.com/paper/2112.03568/) (2021)
- [Unsupervised Multi-object Segmentation Using Attention and Soft-argmax](https://mlsanity.com/paper/2205.13271/) (2022)
- [On permutation-invariant neural networks](https://mlsanity.com/paper/2403.17410/) (2024)
- [Recurrent Complex-Weighted Autoencoders for Unsupervised Object Discovery](https://mlsanity.com/paper/2405.17283/) (2024)
- [Differentiable Mathematical Programming for Object-Centric Representation Learning](https://mlsanity.com/paper/2210.02159/) (2022)
- [RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-centric Learning](https://mlsanity.com/paper/2308.14899/) (2023)
- [Unsupervised Object-Centric Learning from Multiple Unspecified Viewpoints](https://mlsanity.com/paper/2401.01922/) (2024)
- [How does the primate brain combine generative and discriminative computations in vision?](https://mlsanity.com/paper/2401.06005/) (2024)
- [Unsupervised Object Discovery: A Comprehensive Survey and Unified Taxonomy](https://mlsanity.com/paper/2411.00868/) (2024)
- [Compositional Law Parsing with Latent Random Functions](https://mlsanity.com/paper/2209.09115/) (2022)
- [Time-Conditioned Generative Modeling of Object-Centric Representations for Video Decomposition and Prediction](https://mlsanity.com/paper/2301.08951/) (2023)
- [The Learnable Typewriter: A Generative Approach to Text Analysis](https://mlsanity.com/paper/2302.01660/) (2023)
- [OCTScenes: A Versatile Real-World Dataset of Tabletop Scenes for Object-Centric Learning](https://mlsanity.com/paper/2306.09682/) (2023)
- [VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE](https://mlsanity.com/paper/2401.11110/) (2024)
- [Causality-Driven Audits of Model Robustness](https://mlsanity.com/paper/2410.23494/) (2024)
- [CoLa: Chinese Character Decomposition with Compositional Latent Components](https://mlsanity.com/paper/2506.03798/) (2025)
- [Towards Improving the Generation Quality of Autoregressive Slot VAEs](https://mlsanity.com/paper/2206.01370/) (2022)
- [Learning Explicit Object-Centric Representations with Vision Transformers](https://mlsanity.com/paper/2210.14139/) (2022)

## Abstract (arXiv)

Unsupervised multi-object representation learning depends on inductive biases to guide the discovery of object-centric representations that generalize. However, we observe that methods for learning these representations are either impractical due to long training times and large memory consumption or forego key inductive biases. In this work, we introduce EfficientMORL, an efficient framework for the unsupervised learning of object-centric representations. We show that optimization challenges caused by requiring both symmetry and disentanglement can in fact be addressed by high-cost iterative amortized inference by designing the framework to minimize its dependence on it. We take a two-stage approach to inference: first, a hierarchical variational autoencoder extracts symmetric and disentangled representations through bottom-up inference, and second, a lightweight network refines the representations with top-down feedback. The number of refinement steps taken during training is reduced following a curriculum, so that at test time with zero steps the model achieves 99.1% of the refined decomposition performance. We demonstrate strong object decomposition and disentanglement on the standard multi-object benchmark while achieving nearly an order of magnitude faster training and test time inference over the previous state-of-the-art model.