# My House, My Rules: Learning Tidying Preferences with Graph Neural Networks

arXiv: https://arxiv.org/abs/2111.03112 · HTML: https://mlsanity.com/paper/2111.03112/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 4 Nov 2021.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [Semantic Image Inpainting with Deep Generative Models](https://mlsanity.com/paper/1607.07539/) (2016)
- [Rearrangement: A Challenge for Embodied AI](https://mlsanity.com/paper/2011.01975/) (2020)
- [Fusion++: Volumetric Object-Level SLAM](https://mlsanity.com/paper/1808.08378/) (2018)
- [Denoising Criterion for Variational Auto-Encoding Framework](https://mlsanity.com/paper/1511.06406/) (2015)
- [Autonomous Planning Based on Spatial Concepts to Tidy Up Home Environments with Service Robots](https://mlsanity.com/paper/2002.03671/) (2020)

## Built on by (most cited first, 27 of 27)

- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics](https://mlsanity.com/paper/2210.02438/) (2022)
- [Housekeep: Tidying Virtual Households using Commonsense Reasoning](https://mlsanity.com/paper/2205.10712/) (2022)
- [IFOR: Iterative Flow Minimization for Robotic Object Rearrangement](https://mlsanity.com/paper/2202.00732/) (2022)
- [Instant Policy: In-Context Imitation Learning via Graph Diffusion](https://mlsanity.com/paper/2411.12633/) (2024)
- [Graph Neural Networks in IoT: A Survey](https://mlsanity.com/paper/2203.15935/) (2022)
- [Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization](https://mlsanity.com/paper/2505.16348/) (2025)
- [Learning Personalized Agents from Human Feedback](https://mlsanity.com/paper/2602.16173/) (2026)
- [Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models](https://mlsanity.com/paper/2312.04533/) (2023)
- [Transformers are Adaptable Task Planners](https://mlsanity.com/paper/2207.02442/) (2022)
- [Few-Shot In-Context Imitation Learning via Implicit Graph Alignment](https://mlsanity.com/paper/2310.12238/) (2023)
- [TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specification](https://mlsanity.com/paper/2209.00853/) (2022)
- [A Survey on Intelligent Internet of Things: Applications, Security, Privacy, and Future Directions](https://mlsanity.com/paper/2406.03820/) (2024)
- [Where To Start? Transferring Simple Skills to Complex Environments](https://mlsanity.com/paper/2212.06111/) (2022)
- ["Set It Up!": Functional Object Arrangement with Compositional Generative Models](https://mlsanity.com/paper/2405.11928/) (2024)
- [LVDiffusor: Distilling Functional Rearrangement Priors from Large Models into Diffusor](https://mlsanity.com/paper/2312.01474/) (2023)
- [ADAPT: Actively Discovering and Adapting to Preferences for any Task](https://mlsanity.com/paper/2504.04040/) (2025)
- [Graph learning in robotics: a survey](https://mlsanity.com/paper/2310.04294/) (2023)
- [SceneScore: Learning a Cost Function for Object Arrangement](https://mlsanity.com/paper/2311.08530/) (2023)
- [Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning](https://mlsanity.com/paper/2211.15339/) (2022)
- [Robot Behavior Personalization from Sparse User Feedback](https://mlsanity.com/paper/2410.19219/) (2024)
- [Effective Baselines for Multiple Object Rearrangement Planning in Partially Observable Mapped Environments](https://mlsanity.com/paper/2301.09854/) (2023)
- [Tidiness Score-Guided Monte Carlo Tree Search for Visual Tabletop Rearrangement](https://mlsanity.com/paper/2502.17235/) (2025)
- [Personalized Robotic Object Rearrangement from Scene Context](https://mlsanity.com/paper/2505.11108/) (2025)
- [GOPLA: Generalizable Object Placement Learning via Synthetic Augmentation of Human Arrangement](https://mlsanity.com/paper/2510.14627/) (2025)
- [Knolling Bot: Teaching Robots the Human Notion of Tidiness](https://mlsanity.com/paper/2310.04566/) (2023)
- [Explaining Why Things Go Where They Go: Interpretable Constructs of Human Organizational Preferences](https://mlsanity.com/paper/2512.24829/) (2025)

## Abstract (arXiv)

Robots that arrange household objects should do so according to the user's preferences, which are inherently subjective and difficult to model. We present NeatNet: a novel Variational Autoencoder architecture using Graph Neural Network layers, which can extract a low-dimensional latent preference vector from a user by observing how they arrange scenes. Given any set of objects, this vector can then be used to generate an arrangement which is tailored to that user's spatial preferences, with word embeddings used for generalisation to new objects. We develop a tidying simulator to gather rearrangement examples from 75 users, and demonstrate empirically that our method consistently produces neat and personalised arrangements across a variety of rearrangement scenarios.