# Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapy

arXiv: https://arxiv.org/abs/1802.01186 · HTML: https://mlsanity.com/paper/1802.01186/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Feb 2018.

## Built on by (most cited first, 29 of 29)

- [Dawn of the transformer era in speech emotion recognition: closing the valence gap](https://mlsanity.com/paper/2203.07378/) (2022)
- [ChildPlay: A New Benchmark for Understanding Children's Gaze Behaviour](https://mlsanity.com/paper/2307.01630/) (2023)
- [Multimodal Engagement Analysis from Facial Videos in the Classroom](https://mlsanity.com/paper/2101.04215/) (2021)
- [Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questions](https://mlsanity.com/paper/2404.11023/) (2024)
- [A Review of and Roadmap for Data Science and Machine Learning for the Neuropsychiatric Phenotype of Autism](https://mlsanity.com/paper/2303.03577/) (2023)
- [Federated Learning of Shareable Bases for Personalization-Friendly Image Classification](https://mlsanity.com/paper/2304.07882/) (2023)
- [DeepFN: Towards Generalizable Facial Action Unit Recognition with Deep Face Normalization](https://mlsanity.com/paper/2103.02484/) (2021)
- [Federated Continual Learning for Socially Aware Robotics](https://mlsanity.com/paper/2201.05527/) (2022)
- [Automatic Context-Driven Inference of Engagement in HMI: A Survey](https://mlsanity.com/paper/2209.15370/) (2022)
- [Beyond One-Size-Fits-All: A Survey of Personalized Affective Computing in Human-Agent Interaction](https://mlsanity.com/paper/2304.00377/) (2023)
- [Exploring speaker enrolment for few-shot personalisation in emotional vocalisation prediction](https://mlsanity.com/paper/2206.06680/) (2022)
- [Enrolment-based personalisation for improving individual-level fairness in speech emotion recognition](https://mlsanity.com/paper/2406.06665/) (2024)
- [Preserving Privacy in Personalized Models for Distributed Mobile Services](https://mlsanity.com/paper/2101.05855/) (2021)
- [Towards Intercultural Affect Recognition: Audio-Visual Affect Recognition in the Wild Across Six Cultures](https://mlsanity.com/paper/2208.00344/) (2022)
- [Joint Engagement Classification using Video Augmentation Techniques for Multi-person Human-robot Interaction](https://mlsanity.com/paper/2212.14128/) (2022)
- [Causal Stories from Sensor Traces: Auditing Epistemic Overreach in LLM-Generated Personal Sensing Explanations](https://mlsanity.com/paper/2605.08590/) (2026)
- [Human-mediated Large Language Models for Robotic Intervention in Children with Autism Spectrum Disorders](https://mlsanity.com/paper/2402.00260/) (2024)
- [Improving Personalisation in Valence and Arousal Prediction using Data Augmentation](https://mlsanity.com/paper/2404.09042/) (2024)
- [Comparative Analysis of Personalized Voice Activity Detection Systems: Assessing Real-World Effectiveness](https://mlsanity.com/paper/2406.09443/) (2024)
- [Mind the Motions: Benchmarking Theory-of-Mind in Everyday Body Language](https://mlsanity.com/paper/2511.15887/) (2025)
- [Retrieval-Augmented Personalization with Foundation Models for Wearable Stress Detection](https://mlsanity.com/paper/2606.24985/) (2026)
- [HUI360: A 360deg Egocentric Dataset and Baselines for Human-Robot Interaction Anticipation](https://mlsanity.com/paper/2608.11051/) (2026)
- [CACTUS: Detecting and Resolving Conflicts in Objective Functions](https://mlsanity.com/paper/2103.07805/) (2021)
- [Targeted Deep Learning: Framework, Methods, and Applications](https://mlsanity.com/paper/2105.14052/) (2021)
- [Towards Personalized Intelligence at Scale](https://mlsanity.com/paper/2203.06668/) (2022)
- [AdaptoML-UX: An Adaptive User-centered GUI-based AutoML Toolkit for Non-AI Experts and HCI Researchers](https://mlsanity.com/paper/2410.17469/) (2024)
- [A Hybrid Deep Learning Framework for Emotion Recognition in Children with Autism During NAO Robot-Mediated Interaction](https://mlsanity.com/paper/2512.12208/) (2025)
- [LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition](https://mlsanity.com/paper/2605.19821/) (2026)
- [A systematic review of machine learning techniques to address diagnosis and treatment of autism: challenges and opportunities](https://mlsanity.com/paper/2608.18188/) (2026)

## Abstract (arXiv)

Robots have great potential to facilitate future therapies for children on the autism spectrum. However, existing robots lack the ability to automatically perceive and respond to human affect, which is necessary for establishing and maintaining engaging interactions. Moreover, their inference challenge is made harder by the fact that many individuals with autism have atypical and unusually diverse styles of expressing their affective-cognitive states. To tackle the heterogeneity in behavioral cues of children with autism, we use the latest advances in deep learning to formulate a personalized machine learning (ML) framework for automatic perception of the childrens affective states and engagement during robot-assisted autism therapy. The key to our approach is a novel shift from the traditional ML paradigm - instead of using 'one-size-fits-all' ML models, our personalized ML framework is optimized for each child by leveraging relevant contextual information (demographics and behavioral assessment scores) and individual characteristics of each child. We designed and evaluated this framework using a dataset of multi-modal audio, video and autonomic physiology data of 35 children with autism (age 3-13) and from 2 cultures (Asia and Europe), participating in a 25-minute child-robot interaction (~500k datapoints). Our experiments confirm the feasibility of the robot perception of affect and engagement, showing clear improvements due to the model personalization. The proposed approach has potential to improve existing therapies for autism by offering more efficient monitoring and summarization of the therapy progress.