# Dialogue Object Search

arXiv: https://arxiv.org/abs/2107.10653 · HTML: https://mlsanity.com/paper/2107.10653/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 22 Jul 2021.

## Builds on (most cited first, 12 of 12)

- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects](https://mlsanity.com/paper/2006.13171/) (2020)
- [IQA: Visual Question Answering in Interactive Environments](https://mlsanity.com/paper/1712.03316/) (2017)
- [Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning](https://mlsanity.com/paper/1812.00971/) (2018)
- [Talk the Walk: Navigating New York City through Grounded Dialogue](https://mlsanity.com/paper/1807.03367/) (2018)
- [On Evaluating and Comparing Open Domain Dialog Systems](https://mlsanity.com/paper/1801.03625/) (2018)
- [Multi-Resolution POMDP Planning for Multi-Object Search in 3D](https://mlsanity.com/paper/2005.02878/) (2020)
- [Spatial Language Understanding for Object Search in Partially Observed City-scale Environments](https://mlsanity.com/paper/2012.02705/) (2020)
- [Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph](https://mlsanity.com/paper/2103.01350/) (2021)

## Abstract (arXiv)

We envision robots that can collaborate and communicate seamlessly with humans. It is necessary for such robots to decide both what to say and how to act, while interacting with humans. To this end, we introduce a new task, dialogue object search: A robot is tasked to search for a target object (e.g. fork) in a human environment (e.g., kitchen), while engaging in a "video call" with a remote human who has additional but inexact knowledge about the target's location. That is, the robot conducts speech-based dialogue with the human, while sharing the image from its mounted camera. This task is challenging at multiple levels, from data collection, algorithm and system development,to evaluation. Despite these challenges, we believe such a task blocks the path towards more intelligent and collaborative robots. In this extended abstract, we motivate and introduce the dialogue object search task and analyze examples collected from a pilot study. We then discuss our next steps and conclude with several challenges on which we hope to receive feedback.