# Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention

arXiv: https://arxiv.org/abs/2203.06822 · HTML: https://mlsanity.com/paper/2203.06822/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Talk2Car: Taking Control of Your Self-Driving Car](https://mlsanity.com/paper/1909.10838/) (2019)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2005.07310/) (2020)
- [Grounding Human-to-Vehicle Advice for Self-driving Vehicles](https://mlsanity.com/paper/1911.06978/) (2019)
- [Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs](https://mlsanity.com/paper/2011.15124/) (2020)
- [Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/1806.03831/) (2018)
- [Comprehension-guided referring expressions](https://mlsanity.com/paper/1701.03439/) (2017)
- [Training Deeper Neural Machine Translation Models with Transparent Attention](https://mlsanity.com/paper/1808.07561/) (2018)
- [Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning](https://mlsanity.com/paper/2012.14768/) (2020)
- [Improving Grounded Natural Language Understanding through Human-Robot Dialog](https://mlsanity.com/paper/1903.00122/) (2019)
- [Dynamic Layer Aggregation for Neural Machine Translation with Routing-by-Agreement](https://mlsanity.com/paper/1902.05770/) (2019)
- [Giving Commands to a Self-driving Car: A Multimodal Reasoner for Visual Grounding](https://mlsanity.com/paper/2003.08717/) (2020)
- [A Baseline for the Commands For Autonomous Vehicles Challenge](https://mlsanity.com/paper/2004.13822/) (2020)

## Built on by (most cited first, 4 of 4)

- [GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models](https://mlsanity.com/paper/2312.03543/) (2023)
- [Think Before You Drive: World Model-Inspired Multimodal Grounding for Autonomous Vehicles](https://mlsanity.com/paper/2512.03454/) (2025)
- [E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving](https://mlsanity.com/paper/2512.04733/) (2025)
- [Adversarial Testing for Visual Grounding via Image-Aware Property Reduction](https://mlsanity.com/paper/2403.01118/) (2024)

## Abstract (arXiv)

Grounding a command to the visual environment is an essential ingredient for interactions between autonomous vehicles and humans. In this work, we study the problem of language grounding for autonomous vehicles, which aims to localize a region in a visual scene according to a natural language command from a passenger. Prior work only employs the top layer representations of a vision-and-language pre-trained model to predict the region referred to by the command. However, such a method omits the useful features encoded in other layers, and thus results in inadequate understanding of the input scene and command. To tackle this limitation, we present the first layer fusion approach for this task. Since different visual regions may require distinct types of features to disambiguate them from each other, we further propose the region-specific dynamic (RSD) layer attention to adaptively fuse the multimodal information across layers for each region. Extensive experiments on the Talk2Car benchmark demonstrate that our approach helps predict more accurate regions and outperforms state-of-the-art methods.