# LLM-enhanced Scene Graph Learning for Household Rearrangement arXiv: https://arxiv.org/abs/2408.12093 ยท HTML: https://mlsanity.com/paper/2408.12093/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Builds on 38 of its 71 references that are in the map. - First posted on arXiv 22 Aug 2024. ## Builds on (most cited first, 30 of 38) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [ScanQA: 3D Question Answering for Spatial Scene Understanding](https://mlsanity.com/paper/2112.10482/) (2021) - [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023) - [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023) - [Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation](https://mlsanity.com/paper/2306.11290/) (2023) - [Rearrangement: A Challenge for Embodied AI](https://mlsanity.com/paper/2011.01975/) (2020) - [Grounded Human-Object Interaction Hotspots from Video](https://mlsanity.com/paper/1812.04558/) (2018) - [Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes](https://mlsanity.com/paper/2308.08769/) (2023) - [Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detection](https://mlsanity.com/paper/2405.10300/) (2024) - [EGO-TOPO: Environment Affordances from Egocentric Video](https://mlsanity.com/paper/2001.04583/) (2020) - [Visual Room Rearrangement](https://mlsanity.com/paper/2103.16544/) (2021) - [ManipulaTHOR: A Framework for Visual Object Manipulation](https://mlsanity.com/paper/2104.11213/) (2021) - [3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding](https://mlsanity.com/paper/2103.16397/) (2021) - [OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data](https://mlsanity.com/paper/2311.02873/) (2023) - [Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance](https://mlsanity.com/paper/2403.18036/) (2024) - [Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments](https://mlsanity.com/paper/1903.05690/) (2019) - [StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects](https://mlsanity.com/paper/2110.10189/) (2021) - [Housekeep: Tidying Virtual Households using Commonsense Reasoning](https://mlsanity.com/paper/2205.10712/) (2022) - [Putting People in Their Place: Affordance-Aware Human Insertion into Scenes](https://mlsanity.com/paper/2304.14406/) (2023) - [SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments](https://mlsanity.com/paper/2309.04077/) (2023) - [StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects](https://mlsanity.com/paper/2211.04604/) (2022) - [Synthesizing Training Data for Object Detection in Indoor Scenes](https://mlsanity.com/paper/1702.07836/) (2017) - [O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning](https://mlsanity.com/paper/2106.15087/) (2021) - [3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification](https://mlsanity.com/paper/2212.00338/) (2022) - [One-Shot Open Affordance Learning with Foundation Models](https://mlsanity.com/paper/2311.17776/) (2023) - [Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation](https://mlsanity.com/paper/2003.06233/) (2020) - [TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priors](https://mlsanity.com/paper/2207.10761/) (2022) - [GRID: Scene-Graph-based Instruction-driven Robotic Task Planning](https://mlsanity.com/paper/2309.07726/) (2023) ## Built on by (most cited first, 12 of 12) - [CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image](https://mlsanity.com/paper/2502.12894/) (2025) - [GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering](https://mlsanity.com/paper/2412.14480/) (2024) - [Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Task](https://mlsanity.com/paper/2409.08811/) (2024) - [OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging](https://mlsanity.com/paper/2503.01309/) (2025) - [Humanoid World Models: Open World Foundation Models for Humanoid Robotics](https://mlsanity.com/paper/2506.01182/) (2025) - [GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answering](https://mlsanity.com/paper/2506.01174/) (2025) - [Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agents](https://mlsanity.com/paper/2605.12620/) (2026) - [P3: Toward Versatile Embodied Agents](https://mlsanity.com/paper/2508.07033/) (2025) - [Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model](https://mlsanity.com/paper/2510.11509/) (2025) - [Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrieval](https://mlsanity.com/paper/2604.15951/) (2026) - [Grounding Vision and Language to 3D Masks for Long-Horizon Box Rearrangement](https://mlsanity.com/paper/2603.23676/) (2026) - [SPHERE: Adaptive VR Indoor Scene Generation via LLM-Enhanced Spatial Preference Learning and Human-in-the-Loop RL](https://mlsanity.com/paper/2610.02023/) (2026) ## Abstract (arXiv) The household rearrangement task involves spotting misplaced objects in a scene and accommodate them with proper places. It depends both on common-sense knowledge on the objective side and human user preference on the subjective side. In achieving such task, we propose to mine object functionality with user preference alignment directly from the scene itself, without relying on human intervention. To do so, we work with scene graph representation and propose LLM-enhanced scene graph learning which transforms the input scene graph into an affordance-enhanced graph (AEG) with information-enhanced nodes and newly discovered edges (relations). In AEG, the nodes corresponding to the receptacle objects are augmented with context-induced affordance which encodes what kind of carriable objects can be placed on it. New edges are discovered with newly discovered non-local relations. With AEG, we perform task planning for scene rearrangement by detecting misplaced carriables and determining a proper placement for each of them. We test our method by implementing a tiding robot in simulator and perform evaluation on a new benchmark we build. Extensive evaluations demonstrate that our method achieves state-of-the-art performance on misplacement detection and the following rearrangement planning.