# Personalized Robotic Object Rearrangement from Scene Context

arXiv: https://arxiv.org/abs/2505.11108 · HTML: https://mlsanity.com/paper/2505.11108/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 10 of 10)

- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Rearrangement: A Challenge for Embodied AI](https://mlsanity.com/paper/2011.01975/) (2020)
- [Housekeep: Tidying Virtual Households using Commonsense Reasoning](https://mlsanity.com/paper/2205.10712/) (2022)
- [TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priors](https://mlsanity.com/paper/2207.10761/) (2022)
- [My House, My Rules: Learning Tidying Preferences with Graph Neural Networks](https://mlsanity.com/paper/2111.03112/) (2021)
- [APRICOT: Active Preference Learning and Constraint-Aware Task Planning with LLMs](https://mlsanity.com/paper/2410.19656/) (2024)
- [Seeing the Unseen: Visual Common Sense for Semantic Placement](https://mlsanity.com/paper/2401.07770/) (2024)
- [ConSOR: A Context-Aware Semantic Object Rearrangement Framework for Partially Arranged Scenes](https://mlsanity.com/paper/2310.00371/) (2023)
- [Bootstrapping Linear Models for Fast Online Adaptation in Human-Agent Collaboration](https://mlsanity.com/paper/2404.10733/) (2024)
- [DegustaBot: Zero-Shot Visual Preference Estimation for Personalized Multi-Object Rearrangement](https://mlsanity.com/paper/2407.08876/) (2024)

## Built on by (most cited first, 1 of 1)

- [Explaining Why Things Go Where They Go: Interpretable Constructs of Human Organizational Preferences](https://mlsanity.com/paper/2512.24829/) (2025)

## Abstract (arXiv)

Object rearrangement is a key task for household robots requiring personalization without explicit instructions, meaningful object placement in environments occupied with objects, and generalization to unseen objects and new environments. To facilitate research addressing these challenges, we introduce PARSEC, an object rearrangement benchmark for learning user organizational preferences from observed scene context to place objects in a partially arranged environment. PARSEC is built upon a novel dataset of 110K rearrangement examples crowdsourced from 72 users, featuring 93 object categories and 15 environments. To better align with real-world organizational habits, we propose ContextSortLM, an LLM-based personalized rearrangement model that handles flexible user preferences by explicitly accounting for objects with multiple valid placement locations when placing items in partially arranged environments. We evaluate ContextSortLM and existing personalized rearrangement approaches on the PARSEC benchmark and complement these findings with a crowdsourced evaluation of 108 online raters ranking model predictions based on alignment with user preferences. Our results indicate that personalized rearrangement models leveraging multiple scene context sources perform better than models relying on a single context source. Moreover, ContextSortLM outperforms other models in placing objects to replicate the target user's arrangement and ranks among the top two in all three environment categories, as rated by online evaluators. Importantly, our evaluation highlights challenges associated with modeling environment semantics across different environment categories and provides recommendations for future work.