# You Have a Point There: Object Selection Inside an Automobile Using Gaze, Head Pose and Finger Pointing

arXiv: https://arxiv.org/abs/2012.13449 · HTML: https://mlsanity.com/paper/2012.13449/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle](https://mlsanity.com/paper/2111.02327/) (2021)
- [Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects](https://mlsanity.com/paper/2107.12167/) (2021)
- [Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces](https://mlsanity.com/paper/2211.03539/) (2022)
- [What's on your mind? A Mental and Perceptual Load Estimation Framework towards Adaptive In-vehicle Interaction while Driving](https://mlsanity.com/paper/2208.05564/) (2022)
- [It's all about you: Personalized in-Vehicle Gesture Recognition with a Time-of-Flight Camera](https://mlsanity.com/paper/2310.01659/) (2023)
- [Looking for a better fit? An Incremental Learning Multimodal Object Referencing Framework adapting to Individual Drivers](https://mlsanity.com/paper/2401.16123/) (2024)
- [Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle](https://mlsanity.com/paper/2202.07360/) (2022)
- [Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systems](https://mlsanity.com/paper/2309.05787/) (2023)
- [Speak to the City: Multimodal Resolution for Outside-the-Vehicle References](https://mlsanity.com/paper/2609.14691/) (2026)

## Abstract (arXiv)

Sophisticated user interaction in the automotive industry is a fast emerging topic. Mid-air gestures and speech already have numerous applications for driver-car interaction. Additionally, multimodal approaches are being developed to leverage the use of multiple sensors for added advantages. In this paper, we propose a fast and practical multimodal fusion method based on machine learning for the selection of various control modules in an automotive vehicle. The modalities taken into account are gaze, head pose and finger pointing gesture. Speech is used only as a trigger for fusion. Single modality has previously been used numerous times for recognition of the user's pointing direction. We, however, demonstrate how multiple inputs can be fused together to enhance the recognition performance. Furthermore, we compare different deep neural network architectures against conventional Machine Learning methods, namely Support Vector Regression and Random Forests, and show the enhancements in the pointing direction accuracy using deep learning. The results suggest a great potential for the use of multimodal inputs that can be applied to more use cases in the vehicle.