# Zero-shot Active Visual Search (ZAVIS): Intelligent Object Search for Robotic Assistants

arXiv: https://arxiv.org/abs/2209.08803 · HTML: https://mlsanity.com/paper/2209.08803/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 19 Sep 2022.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [Simple but Effective: CLIP Embeddings for Embodied AI](https://mlsanity.com/paper/2111.09888/) (2021)
- [Grad-CAM: Why did you say that?](https://mlsanity.com/paper/1611.07450/) (2016)
- [Zero-shot object goal visual navigation](https://mlsanity.com/paper/2206.07423/) (2022)
- [Semantic Linking Maps for Active Visual Object Search](https://mlsanity.com/paper/2006.10807/) (2020)
- [Towards Optimal Correlational Object Search](https://mlsanity.com/paper/2110.09991/) (2021)
- [Task Agnostic Robust Learning on Corrupt Outputs by Correlation-Guided Mixture Density Networks](https://mlsanity.com/paper/1805.06431/) (2018)
- [Elucidating Robust Learning with Uncertainty-Aware Corruption Pattern Estimation](https://mlsanity.com/paper/2111.01632/) (2021)

## Built on by (most cited first, 7 of 7)

- [VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Model](https://mlsanity.com/paper/2401.02695/) (2024)
- [Zero-shot Object Navigation with Vision-Language Models Reasoning](https://mlsanity.com/paper/2410.18570/) (2024)
- [Multiple Thinking Achieving Meta-Ability Decoupling for Object Navigation](https://mlsanity.com/paper/2302.01520/) (2023)
- [CAMON: Cooperative Agents for Multi-Object Navigation with LLM-based Conversations](https://mlsanity.com/paper/2407.00632/) (2024)
- [Large-Language-Model-Guided State Estimation for Partially Observable Task and Motion Planning](https://mlsanity.com/paper/2603.03704/) (2026)
- [SOCRATES: Text-based Human Search and Approach using a Robot Dog](https://mlsanity.com/paper/2302.05324/) (2023)
- [SR-Nav: Spatial Relationships Matter for Zero-shot Object Goal Navigation](https://mlsanity.com/paper/2603.18443/) (2026)

## Abstract (arXiv)

In this paper, we focus on the problem of efficiently locating a target object described with free-form language using a mobile robot equipped with vision sensors (e.g., an RGBD camera). Conventional active visual search predefines a set of objects to search for, rendering these techniques restrictive in practice. To provide added flexibility in active visual searching, we propose a system where a user can enter target commands using free-form language; we call this system Active Visual Search in the Wild (AVSW). AVSW detects and plans to search for a target object inputted by a user through a semantic grid map represented by static landmarks (e.g., desk or bed). For efficient planning of object search patterns, AVSW considers commonsense knowledge-based co-occurrence and predictive uncertainty while deciding which landmarks to visit first. We validate the proposed method with respect to SR (success rate) and SPL (success weighted by path length) in both simulated and real-world environments. The proposed method outperforms previous methods in terms of SPL in simulated scenarios with an average gap of 0.283. We further demonstrate AVSW with a Pioneer-3AT robot in real-world studies.