# Few-Shot Visual Grounding for Natural Human-Robot Interaction

arXiv: https://arxiv.org/abs/2103.09720 · HTML: https://mlsanity.com/paper/2103.09720/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 17 Mar 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [12-in-1: Multi-Task Vision and Language Representation Learning](https://mlsanity.com/paper/1912.02315/) (2019)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [Zero-Shot Grounding of Objects from Natural Language Queries](https://mlsanity.com/paper/1908.07129/) (2019)
- [Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/1806.03831/) (2018)
- [Conditional Image-Text Embedding Networks](https://mlsanity.com/paper/1711.08389/) (2017)
- [Learning Cross-modal Context Graph for Visual Grounding](https://mlsanity.com/paper/1911.09042/) (2019)
- [Query-guided Regression Network with Context Policy for Phrase Grounding](https://mlsanity.com/paper/1708.01676/) (2017)
- [Improving Grounded Natural Language Understanding through Human-Robot Dialog](https://mlsanity.com/paper/1903.00122/) (2019)

## Built on by (most cited first, 2 of 2)

- [R2-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations](https://mlsanity.com/paper/2403.04924/) (2024)
- [Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding](https://mlsanity.com/paper/2505.15123/) (2025)

## Abstract (arXiv)

Natural Human-Robot Interaction (HRI) is one of the key components for service robots to be able to work in human-centric environments. In such dynamic environments, the robot needs to understand the intention of the user to accomplish a task successfully. Towards addressing this point, we propose a software architecture that segments a target object from a crowded scene, indicated verbally by a human user. At the core of our system, we employ a multi-modal deep neural network for visual grounding. Unlike most grounding methods that tackle the challenge using pre-trained object detectors via a two-stepped process, we develop a single stage zero-shot model that is able to provide predictions in unseen data. We evaluate the performance of the proposed model on real RGB-D data collected from public scene datasets. Experimental results showed that the proposed model performs well in terms of accuracy and speed, while showcasing robustness to variation in the natural language input.