# Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systems

arXiv: https://arxiv.org/abs/2309.05787 · HTML: https://mlsanity.com/paper/2309.05787/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 58 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 14 of 14)

- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [Logic Tensor Networks for Semantic Image Interpretation](https://mlsanity.com/paper/1705.08968/) (2017)
- [Classical Planning in Deep Latent Space: Bridging the Subsymbolic-Symbolic Boundary](https://mlsanity.com/paper/1705.00154/) (2017)
- [Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization](https://mlsanity.com/paper/2202.10341/) (2022)
- [The EMPATHIC Framework for Task Learning from Implicit Human Feedback](https://mlsanity.com/paper/2009.13649/) (2020)
- [You Have a Point There: Object Selection Inside an Automobile Using Gaze, Head Pose and Finger Pointing](https://mlsanity.com/paper/2012.13449/) (2020)
- [ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle](https://mlsanity.com/paper/2111.02327/) (2021)
- [Studying Person-Specific Pointing and Gaze Behavior for Multimodal Referencing of Outside Objects from a Moving Vehicle](https://mlsanity.com/paper/2009.11195/) (2020)
- [Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects](https://mlsanity.com/paper/2107.12167/) (2021)
- [Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces](https://mlsanity.com/paper/2211.03539/) (2022)
- [What's on your mind? A Mental and Perceptual Load Estimation Framework towards Adaptive In-vehicle Interaction while Driving](https://mlsanity.com/paper/2208.05564/) (2022)
- [In-Vehicle Interface Adaptation to Environment-Induced Cognitive Workload](https://mlsanity.com/paper/2210.11271/) (2022)

## Abstract (arXiv)

Recent advances in machine learning, particularly deep learning, have enabled autonomous systems to perceive and comprehend objects and their environments in a perceptual subsymbolic manner. These systems can now perform object detection, sensor data fusion, and language understanding tasks. However, there is a growing need to enhance these systems to understand objects and their environments more conceptually and symbolically. It is essential to consider both the explicit teaching provided by humans (e.g., describing a situation or explaining how to act) and the implicit teaching obtained by observing human behavior (e.g., through the system's sensors) to achieve this level of powerful artificial intelligence. Thus, the system must be designed with multimodal input and output capabilities to support implicit and explicit interaction models. In this position paper, we argue for considering both types of inputs, as well as human-in-the-loop and incremental learning techniques, for advancing the field of artificial intelligence and enabling autonomous systems to learn like humans. We propose several hypotheses and design guidelines and highlight a use case from related work to achieve this goal.