# A Simple Approach for Visual Rearrangement: 3D Mapping and Semantic Search

arXiv: https://arxiv.org/abs/2206.13396 · HTML: https://mlsanity.com/paper/2206.13396/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 21 Jun 2022.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects](https://mlsanity.com/paper/2006.13171/) (2020)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [Neural Topological SLAM for Visual Navigation](https://mlsanity.com/paper/2005.12256/) (2020)
- [Episodic Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2105.06453/) (2021)
- [IQA: Visual Question Answering in Interactive Environments](https://mlsanity.com/paper/1712.03316/) (2017)
- [Rearrangement: A Challenge for Embodied AI](https://mlsanity.com/paper/2011.01975/) (2020)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [Visual Semantic Navigation using Scene Priors](https://mlsanity.com/paper/1810.06543/) (2018)
- [FILM: Following Instructions in Language with Modular Methods](https://mlsanity.com/paper/2110.07342/) (2021)
- [Visual Room Rearrangement](https://mlsanity.com/paper/2103.16544/) (2021)
- [A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution](https://mlsanity.com/paper/2107.05612/) (2021)
- [Bayesian Relational Memory for Semantic Visual Navigation](https://mlsanity.com/paper/1909.04306/) (2019)
- [SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency](https://mlsanity.com/paper/2112.01001/) (2021)
- [Co-Fusion: Real-time Segmentation, Tracking and Fusion of Multiple Objects](https://mlsanity.com/paper/1706.06629/) (2017)
- [IFOR: Iterative Flow Minimization for Robotic Object Rearrangement](https://mlsanity.com/paper/2202.00732/) (2022)
- [Analysis and Observations from the First Amazon Picking Challenge](https://mlsanity.com/paper/1601.05484/) (2016)
- [A Behavioral Approach to Visual Navigation with Graph Localization Networks](https://mlsanity.com/paper/1903.00445/) (2019)
- [Continuous Scene Representations for Embodied AI](https://mlsanity.com/paper/2203.17251/) (2022)
- [NeRP: Neural Rearrangement Planning for Unknown Objects](https://mlsanity.com/paper/2106.01352/) (2021)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [Factorizing Perception and Policy for Interactive Instruction Following](https://mlsanity.com/paper/2012.03208/) (2020)

## Built on by (most cited first, 6 of 6)

- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments](https://mlsanity.com/paper/2604.22409/) (2026)
- [Hierarchical Object-Oriented POMDP Planning for Object Rearrangement](https://mlsanity.com/paper/2412.01348/) (2024)
- [AI-Driven Relocation Tracking in Dynamic Kitchen Environments](https://mlsanity.com/paper/2503.01547/) (2025)
- [Explaining Why Things Go Where They Go: Interpretable Constructs of Human Organizational Preferences](https://mlsanity.com/paper/2512.24829/) (2025)

## Abstract (arXiv)

Physically rearranging objects is an important capability for embodied agents. Visual room rearrangement evaluates an agent's ability to rearrange objects in a room to a desired goal based solely on visual input. We propose a simple yet effective method for this problem: (1) search for and map which objects need to be rearranged, and (2) rearrange each object until the task is complete. Our approach consists of an off-the-shelf semantic segmentation model, voxel-based semantic map, and semantic search policy to efficiently find objects that need to be rearranged. On the AI2-THOR Rearrangement Challenge, our method improves on current state-of-the-art end-to-end reinforcement learning-based methods that learn visual rearrangement policies from 0.53% correct rearrangement to 16.56%, using only 2.7% as many samples from the environment.