# Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces

arXiv: https://arxiv.org/abs/2211.03539 · HTML: https://mlsanity.com/paper/2211.03539/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 49 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization](https://mlsanity.com/paper/2202.10341/) (2022)
- [The EMPATHIC Framework for Task Learning from Implicit Human Feedback](https://mlsanity.com/paper/2009.13649/) (2020)
- [You Have a Point There: Object Selection Inside an Automobile Using Gaze, Head Pose and Finger Pointing](https://mlsanity.com/paper/2012.13449/) (2020)
- [ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle](https://mlsanity.com/paper/2111.02327/) (2021)
- [Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle](https://mlsanity.com/paper/2009.11195/) (2020)
- [Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects](https://mlsanity.com/paper/2107.12167/) (2021)

## Built on by (most cited first, 4 of 4)

- [It's all about you: Personalized in-Vehicle Gesture Recognition with a Time-of-Flight Camera](https://mlsanity.com/paper/2310.01659/) (2023)
- [Looking for a better fit? An Incremental Learning Multimodal Object Referencing Framework adapting to Individual Drivers](https://mlsanity.com/paper/2401.16123/) (2024)
- [Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systems](https://mlsanity.com/paper/2309.05787/) (2023)
- [AdaptoML-UX: An Adaptive User-centered GUI-based AutoML Toolkit for Non-AI Experts and HCI Researchers](https://mlsanity.com/paper/2410.17469/) (2024)

## Abstract (arXiv)

With the recently increasing capabilities of modern vehicles, novel approaches for interaction emerged that go beyond traditional touch-based and voice command approaches. Therefore, hand gestures, head pose, eye gaze, and speech have been extensively investigated in automotive applications for object selection and referencing. Despite these significant advances, existing approaches mostly employ a one-model-fits-all approach unsuitable for varying user behavior and individual differences. Moreover, current referencing approaches either consider these modalities separately or focus on a stationary situation, whereas the situation in a moving vehicle is highly dynamic and subject to safety-critical constraints. In this paper, I propose a research plan for a user-centered adaptive multimodal fusion approach for referencing external objects from a moving vehicle. The proposed plan aims to provide an open-source framework for user-centered adaptation and personalization using user observations and heuristics, multimodal fusion, clustering, transfer-of-learning for model adaptation, and continuous learning, moving towards trusted human-centered artificial intelligence.