# Giving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding

arXiv: https://arxiv.org/abs/2003.08717 · HTML: https://mlsanity.com/paper/2003.08717/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2020.

## Built on by (most cited first, 5 of 5)

- [GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models](https://mlsanity.com/paper/2312.03543/) (2023)
- [Giving Commands to a Self-Driving Car: How to Deal with Uncertain Situations?](https://mlsanity.com/paper/2106.04232/) (2021)
- [Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention](https://mlsanity.com/paper/2203.06822/) (2022)
- [Estimation of Appearance and Occupancy Information in Birds Eye View from Surround Monocular Images](https://mlsanity.com/paper/2211.04557/) (2022)
- [Adversarial Testing for Visual Grounding via Image-Aware Property Reduction](https://mlsanity.com/paper/2403.01118/) (2024)

## Abstract (arXiv)

We propose a new spatial memory module and a spatial reasoner for the Visual Grounding (VG) task. The goal of this task is to find a certain object in an image based on a given textual query. Our work focuses on integrating the regions of a Region Proposal Network (RPN) into a new multi-step reasoning model which we have named a Multimodal Spatial Region Reasoner (MSRR). The introduced model uses the object regions from an RPN as initialization of a 2D spatial memory and then implements a multi-step reasoning process scoring each region according to the query, hence why we call it a multimodal reasoner. We evaluate this new model on challenging datasets and our experiments show that our model that jointly reasons over the object regions of the image and words of the query largely improves accuracy compared to current state-of-the-art models.