# Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation

arXiv: https://arxiv.org/abs/2207.00787 · HTML: https://mlsanity.com/paper/2207.00787/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 84 references that are in the map.
- First posted on arXiv 2 Jul 2022.

## Builds on (most cited first, 30 of 42)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Truncated Back-propagation for Bilevel Optimization](https://mlsanity.com/paper/1810.10667/) (2018)
- [Recasting Gradient-Based Meta-Learning as Hierarchical Bayes](https://mlsanity.com/paper/1801.08930/) (2018)
- [A Theory of Generative ConvNet](https://mlsanity.com/paper/1602.03264/) (2016)
- [Using Fast Weights to Attend to the Recent Past](https://mlsanity.com/paper/1610.06258/) (2016)
- [SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos](https://mlsanity.com/paper/2206.07764/) (2022)
- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [Neural Expectation Maximization](https://mlsanity.com/paper/1708.03498/) (2017)
- [Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs](https://mlsanity.com/paper/1901.07017/) (2019)
- [Entity Abstraction in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/1910.12827/) (2019)
- [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021)
- [Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions](https://mlsanity.com/paper/1802.10353/) (2018)
- [Iterative Amortized Inference](https://mlsanity.com/paper/1807.09356/) (2018)

## Built on by (most cited first, 30 of 46)

- [Object-Centric Slot Diffusion](https://mlsanity.com/paper/2303.10834/) (2023)
- [Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities](https://mlsanity.com/paper/2306.04829/) (2023)
- [Improving Object-centric Learning with Query Optimization](https://mlsanity.com/paper/2210.08990/) (2022)
- [SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers](https://mlsanity.com/paper/2312.00648/) (2023)
- [Artificial Kuramoto Oscillatory Neurons](https://mlsanity.com/paper/2410.13821/) (2024)
- [Neural Systematic Binder](https://mlsanity.com/paper/2211.01177/) (2022)
- [Improving Cross-Modal Retrieval with Set of Diverse Embeddings](https://mlsanity.com/paper/2211.16761/) (2022)
- [Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learning](https://mlsanity.com/paper/2301.05169/) (2023)
- [TorchDEQ: A Library for Deep Equilibrium Models](https://mlsanity.com/paper/2310.18605/) (2023)
- [Object-centric architectures enable efficient causal representation learning](https://mlsanity.com/paper/2310.19054/) (2023)
- [Interaction Asymmetry: A General Principle for Learning Composable Abstractions](https://mlsanity.com/paper/2411.07784/) (2024)
- [Grounded Object Centric Learning](https://mlsanity.com/paper/2307.09437/) (2023)
- [Unlocking Slot Attention by Changing Optimal Transport Costs](https://mlsanity.com/paper/2301.13197/) (2023)
- [Neurosymbolic Grounding for Compositional World Models](https://mlsanity.com/paper/2310.12690/) (2023)
- [Learning to Compose: Improving Object Centric Learning by Injecting Compositionality](https://mlsanity.com/paper/2405.00646/) (2024)
- [Object-Centric Learning with Slot Mixture Module](https://mlsanity.com/paper/2311.04640/) (2023)
- [MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning](https://mlsanity.com/paper/2505.20772/) (2025)
- [A Fixed-Point Approach to Unified Prompt-Based Counting](https://mlsanity.com/paper/2403.10236/) (2024)
- [On permutation-invariant neural networks](https://mlsanity.com/paper/2403.17410/) (2024)
- [Identifiable Object-Centric Representation Learning via Probabilistic Slot Attention](https://mlsanity.com/paper/2406.07141/) (2024)
- [Sensitivity of Slot-Based Object-Centric Models to their Number of Slots](https://mlsanity.com/paper/2305.18890/) (2023)
- [Spotlight Attention: Robust Object-Centric Learning With a Spatial Locality Prior](https://mlsanity.com/paper/2305.19550/) (2023)
- [Benchmarks for Physical Reasoning AI](https://mlsanity.com/paper/2312.10728/) (2023)
- [Robust and Controllable Object-Centric Learning through Energy-based Models](https://mlsanity.com/paper/2210.05519/) (2022)
- [Concept-Centric Transformers: Enhancing Model Interpretability through Object-Centric Concept Learning within a Shared Global Workspace](https://mlsanity.com/paper/2305.15775/) (2023)
- [Cycle Consistency Driven Object Discovery](https://mlsanity.com/paper/2306.02204/) (2023)
- [Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement](https://mlsanity.com/paper/2303.11373/) (2023)
- [LPGD: A General Framework for Backpropagation through Embedded Optimization Layers](https://mlsanity.com/paper/2407.05920/) (2024)
- [Bootstrapping Top-down Information for Self-modulating Slot Attention](https://mlsanity.com/paper/2411.01801/) (2024)
- [Slot Structured World Models](https://mlsanity.com/paper/2402.03326/) (2024)

## Abstract (arXiv)

Iterative refinement -- start with a random guess, then iteratively improve the guess -- is a useful paradigm for representation learning because it offers a way to break symmetries among equally plausible explanations for the data. This property enables the application of such methods to infer representations of sets of entities, such as objects in physical scenes, structurally resembling clustering algorithms in latent space. However, most prior works differentiate through the unrolled refinement process, which can make optimization challenging. We observe that such methods can be made differentiable by means of the implicit function theorem, and develop an implicit differentiation approach that improves the stability and tractability of training by decoupling the forward and backward passes. This connection enables us to apply advances in optimizing implicit layers to not only improve the optimization of the slot attention module in SLATE, a state-of-the-art method for learning entity representations, but do so with constant space and time complexity in backpropagation and only one additional line of code.