# Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects

arXiv: https://arxiv.org/abs/2107.12167 · HTML: https://mlsanity.com/paper/2107.12167/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 30 references that are in the map.
- First posted on arXiv 26 Jul 2021.

## Builds on (most cited first, 4 of 4)

- [Speak2Label: Using Domain Knowledge for Creating a Large Scale Driver Gaze Zone Estimation Dataset](https://mlsanity.com/paper/2004.05973/) (2020)
- [You Have a Point There: Object Selection Inside an Automobile Using Gaze, Head Pose and Finger Pointing](https://mlsanity.com/paper/2012.13449/) (2020)
- [Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle](https://mlsanity.com/paper/2009.11195/) (2020)
- [Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction](https://mlsanity.com/paper/1709.02251/) (2017)

## Built on by (most cited first, 5 of 5)

- [Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces](https://mlsanity.com/paper/2211.03539/) (2022)
- [Looking for a better fit? An Incremental Learning Multimodal Object Referencing Framework adapting to Individual Drivers](https://mlsanity.com/paper/2401.16123/) (2024)
- [Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle](https://mlsanity.com/paper/2202.07360/) (2022)
- [Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systems](https://mlsanity.com/paper/2309.05787/) (2023)
- [Speak to the City: Multimodal Resolution for Outside-the-Vehicle References](https://mlsanity.com/paper/2609.14691/) (2026)

## Abstract (arXiv)

There is a growing interest in more intelligent natural user interaction with the car. Hand gestures and speech are already being applied for driver-car interaction. Moreover, multimodal approaches are also showing promise in the automotive industry. In this paper, we utilize deep learning for a multimodal fusion network for referencing objects outside the vehicle. We use features from gaze, head pose and finger pointing simultaneously to precisely predict the referenced objects in different car poses. We demonstrate the practical limitations of each modality when used for a natural form of referencing, specifically inside the car. As evident from our results, we overcome the modality specific limitations, to a large extent, by the addition of other modalities. This work highlights the importance of multimodal sensing, especially when moving towards natural user interaction. Furthermore, our user based analysis shows noteworthy differences in recognition of user behavior depending upon the vehicle pose.