mlsanityData to 7 Oct 2026

Visual Room Rearrangement

arXiv 2103.16544 ↗First posted 30 Mar 2021cs.CV cs.ROcited by 101

Not on the map. Why?

Abstract (arXiv)

There has been a significant recent progress in the field of Embodied AI with researchers developing models and algorithms enabling embodied agents to navigate and interact within completely unseen environments. In this paper, we propose a new dataset and baseline models for the task of Rearrangement. We particularly focus on the task of Room Rearrangement: an agent begins by exploring a room and recording objects' initial configurations. We then remove the agent and change the poses and states (e.g., open/closed) of some objects in the room. The agent must restore the initial configurations of all objects in the room. Our dataset, named RoomR, includes 6,000 distinct rearrangement settings involving 72 different object types in 120 scenes. Our experiments show that solving this challenging interactive task that involves navigation and object interaction is beyond the capabilities of the current state-of-the-art techniques for embodied tasks and we are still very far from achieving perfect performance on these types of tasks. The code and the dataset are available at: https://ai2thor.allenai.org/rearrangement

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

36 of 55 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  10. 2018On Evaluation of Embodied Navigation Agentscited by 526
  11. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  12. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
24 more
  1. 2017Embodied Question Answeringcited by 335
  2. 2020Learning to Explore using Active Neural SLAMcited by 308
  3. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  4. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  5. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  6. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  7. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  8. 2018Visual Semantic Navigation using Scene Priorscited by 127
  9. 2019Learning Exploration Policies for Navigationcited by 113
  10. 2020Occupancy Anticipation for Efficient Exploration and Navigationcited by 104
  11. 2018Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learningcited by 91
  12. 2020AllenAct: A Framework for Embodied AI Researchcited by 54
  13. 2020A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Taskscited by 52
  14. 2019Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassemblycited by 51
  15. 2020ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulationcited by 47
  16. 2017Visual Semantic Planning using Deep Successor Representationscited by 45
  17. 2019Two Body Problem: Collaborative Visual Task Completioncited by 45
  18. 2016Analysis and Observations from the First Amazon Picking Challengecited by 41
  19. 2016FFRob: Leveraging Symbolic Planning for Efficient Task and Motion Planningcited by 34
  20. 2019Monte-Carlo Tree Search for Efficient Visually Guided Rearrangement Planningcited by 23
  21. 2020Self-supervised Learning for Precise Pick-and-place without Object Modelcited by 20
  22. 2019Multi-Object Rearrangement with Monte Carlo Tree Search:A Case Study on Planar Nonprehensile Sortingcited by 12
  23. 2017Pick and Place Without Geometric Object Modelscited by 9
  24. 2018Rearrangement with Nonprehensile Manipulation Using Deep Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 24; 2023: 28; 2024: 26; 2025: 10; 2026: 3.

101 papers.

Sort:

By area:

  1. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  2. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  3. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  4. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  5. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  6. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  7. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  8. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  9. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  10. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  11. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  12. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 140
38 more
  1. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  2. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 104
  3. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 100
  4. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 75
  5. 2023OBJECT 3DIT: Language-guided 3D-aware Image Editingcited by 62
  6. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  7. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  8. 2024AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectoriescited by 56
  9. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 55
  10. 2024GRUtopia: Dream General Robots in a City at Scalecited by 54
  11. 2022OPD: Single-view 3D Openable Part Detectioncited by 52
  12. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  13. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 47
  14. 2022Retrospectives on the Embodied AI Workshopcited by 46
  15. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  16. 2022IFOR: Iterative Flow Minimization for Robotic Object Rearrangementcited by 42
  17. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  18. 2022Continuous Scene Representations for Embodied AIcited by 39
  19. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 37
  20. 2022Iterative Vision-and-Language Navigationcited by 34
  21. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  22. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  23. 2024Prioritized Semantic Learning for Zero-shot Instance Navigationcited by 30
  24. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  25. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
  26. 2023Universal Visual Decomposer: Long-Horizon Manipulation Made Easycited by 23
  27. 2021Semantically Grounded Object Matching for Robust Robotic Scene Rearrangementcited by 21
  28. 2023FGPrompt: Fine-grained Goal Prompting for Image-goal Navigationcited by 21
  29. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  30. 2025Grounding Multimodal LLMs to Embodied Agents that Ask for Help with Reinforcement Learningcited by 21
  31. 2021GridToPix: Training Embodied Agents with Minimal Supervisioncited by 20
  32. 2022Learning Active Camera for Multi-Object Navigationcited by 20
  33. 2021Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agentscited by 19
  34. 2023Selective Visual Representations Improve Convergence and Generalization for Embodied AIcited by 19
  35. 2022Ask4Help: Learning to Leverage an Expert for Embodied Taskscited by 18
  36. 2021Shaping embodied agent behavior with activity-context priors from egocentric videocited by 17
  37. 2024OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representationcited by 17
  38. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16

50 most cited of 101 All 101 (JSON)