mlsanityData to 7 Oct 2026

Neural Constraint Satisfaction: Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement

arXiv 2303.11373 ↗First posted 20 Mar 2023cs.LG cs.AI cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Object rearrangement is a challenge for embodied agents because solving these tasks requires generalizing across a combinatorially large set of configurations of entities and their locations. Worse, the representations of these entities are unknown and must be inferred from sensory percepts. We present a hierarchical abstraction approach to uncover these underlying entities and achieve combinatorial generalization from unstructured visual inputs. By constructing a factorized transition graph over clusters of entity representations inferred from pixels, we show how to learn a correspondence between intervening on states of entities in the agent's model and acting on objects in the environment. We use this correspondence to develop a method for control that generalizes to different numbers and configurations of objects, which outperforms current offline deep RL methods when evaluated on simulated rearrangement tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

34 of 57 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  5. 2020Object-Centric Learning with Slot Attentioncited by 865
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  8. 2021Conditional Object-Centric Learning from Videocited by 247
  9. 2020On the Binding Problem in Artificial Neural Networkscited by 230
  10. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  11. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 165
  12. 2020Rearrangement: A Challenge for Embodied AIcited by 153
22 more
  1. 2021Illiterate DALL-E Learns to Composecited by 147
  2. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  3. 2017Neural Expectation Maximizationcited by 123
  4. 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
  5. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  6. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  7. 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
  8. 2019Procedure Planning in Instructional Videoscited by 92
  9. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  10. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
  11. 2018Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstrationcited by 60
  12. 2019Structured agents for physical constructioncited by 50
  13. 2020Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systemscited by 48
  14. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
  15. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 41
  16. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  17. 2020Sparse Graphical Memory for Robust Planningcited by 30
  18. 2020Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulationcited by 25
  19. 2018Composable Planning with Attributescited by 21
  20. 2020Plan2Vec: Unsupervised Representation Learning by Latent Planscited by 21
  21. 2021Self-supervised Reinforcement Learning with Independently Controllable Subgoalscited by 16
  22. 2020Self-Supervised Goal-Conditioned Pick and Placecited by 5

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 2.

6 papers.

  1. 2023Learning Dynamic Attribute-factored World Models for Efficient Multi-object Reinforcement Learningcited by 15
  2. 2023Neurosymbolic Grounding for Compositional World Modelscited by 14
  3. 2023Constraining Generative Models for Engineering Design with Negative Datacited by 11
  4. 2023Towards Combinatorial Generalization for Catalysts: A Kohn-Sham Charge-Density Approachcited by 5
  5. 2024Neural Slot Interpreters: Grounding Object Semantics in Emergent Slot Representationscited by 2
  6. 2024Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstractioncited by 2