# Towards Unified Interactive Visual Grounding in The Wild

arXiv: https://arxiv.org/abs/2401.16699 · HTML: https://mlsanity.com/paper/2401.16699/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jan 2024.

## Built on by (most cited first, 3 of 3)

- [Seeing Beyond Classes: Zero-Shot Grounded Situation Recognition via Language Explainer](https://mlsanity.com/paper/2404.15785/) (2024)
- [Robi Butler: Multimodal Remote Interaction with a Household Robot Assistant](https://mlsanity.com/paper/2409.20548/) (2024)
- [SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interaction](https://mlsanity.com/paper/2402.11792/) (2024)

## Abstract (arXiv)

Interactive visual grounding in Human-Robot Interaction (HRI) is challenging yet practical due to the inevitable ambiguity in natural languages. It requires robots to disambiguate the user input by active information gathering. Previous approaches often rely on predefined templates to ask disambiguation questions, resulting in performance reduction in realistic interactive scenarios. In this paper, we propose TiO, an end-to-end system for interactive visual grounding in human-robot interaction. Benefiting from a unified formulation of visual dialogue and grounding, our method can be trained on a joint of extensive public data, and show superior generality to diversified and challenging open-world scenarios. In the experiments, we validate TiO on GuessWhat?! and InViG benchmarks, setting new state-of-the-art performance by a clear margin. Moreover, we conduct HRI experiments on the carefully selected 150 challenging scenes as well as real-robot platforms. Results show that our method demonstrates superior generality to diversified visual and language inputs with a high success rate. Codes and demos are available at https://github.com/jxu124/TiO.