# A System for Generalized 3D Multi-Object Search

arXiv: https://arxiv.org/abs/2303.03178 · HTML: https://mlsanity.com/paper/2303.03178/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 45 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 18 of 18)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects](https://mlsanity.com/paper/2006.13171/) (2020)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [Visual Semantic Navigation using Scene Priors](https://mlsanity.com/paper/1810.06543/) (2018)
- [Skill Induction and Planning with Latent Language](https://mlsanity.com/paper/2110.01517/) (2021)
- [Visual Navigation with Spatial Attention](https://mlsanity.com/paper/2104.09807/) (2021)
- [Semantic Linking Maps for Active Visual Object Search](https://mlsanity.com/paper/2006.10807/) (2020)
- [Towards Optimal Correlational Object Search](https://mlsanity.com/paper/2110.09991/) (2021)
- [Learning Long-Horizon Robot Exploration Strategies for Multi-Object Search in Continuous Action Spaces](https://mlsanity.com/paper/2205.11384/) (2022)
- [Occlusion-Aware Search for Object Retrieval in Clutter](https://mlsanity.com/paper/2011.03334/) (2020)
- [Multi-Resolution POMDP Planning for Multi-Object Search in 3D](https://mlsanity.com/paper/2005.02878/) (2020)
- [Spatial Language Understanding for Object Search in Partially Observed City-scale Environments](https://mlsanity.com/paper/2012.02705/) (2020)
- [pomdp_py: A Framework to Build and Solve POMDP Problems](https://mlsanity.com/paper/2004.10099/) (2020)
- [POMP++: Pomcp-based Active Visual Search in unknown indoor environments](https://mlsanity.com/paper/2107.00914/) (2021)
- [Generalized Object Search](https://mlsanity.com/paper/2301.10121/) (2023)

## Built on by (most cited first, 9 of 9)

- [Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation](https://mlsanity.com/paper/2409.18313/) (2024)
- [Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulation](https://mlsanity.com/paper/2307.06125/) (2023)
- [LLM-Guided Probabilistic Program Induction for POMDP Model Estimation](https://mlsanity.com/paper/2505.02216/) (2025)
- [Find Everything: A General Vision Language Model Approach to Multi-Object Search](https://mlsanity.com/paper/2410.00388/) (2024)
- [CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graph](https://mlsanity.com/paper/2501.13338/) (2025)
- [Ella: Embodied Social Agents with Lifelong Memory](https://mlsanity.com/paper/2506.24019/) (2025)
- [Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning](https://mlsanity.com/paper/2603.03704/) (2026)
- [Hierarchical Object-Oriented POMDP Planning for Object Rearrangement](https://mlsanity.com/paper/2412.01348/) (2024)
- [Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection](https://mlsanity.com/paper/2607.15974/) (2026)

## Abstract (arXiv)

Searching for objects is a fundamental skill for robots. As such, we expect object search to eventually become an off-the-shelf capability for robots, similar to e.g., object detection and SLAM. In contrast, however, no system for 3D object search exists that generalizes across real robots and environments. In this paper, building upon a recent theoretical framework that exploited the octree structure for representing belief in 3D, we present GenMOS (Generalized Multi-Object Search), the first general-purpose system for multi-object search (MOS) in a 3D region that is robot-independent and environment-agnostic. GenMOS takes as input point cloud observations of the local region, object detection results, and localization of the robot's view pose, and outputs a 6D viewpoint to move to through online planning. In particular, GenMOS uses point cloud observations in three ways: (1) to simulate occlusion; (2) to inform occupancy and initialize octree belief; and (3) to sample a belief-dependent graph of view positions that avoid obstacles. We evaluate our system both in simulation and on two real robot platforms. Our system enables, for example, a Boston Dynamics Spot robot to find a toy cat hidden underneath a couch in under one minute. We further integrate 3D local search with 2D global search to handle larger areas, demonstrating the resulting system in a 25m2 lobby area.