# Learning on the Job: Long-Term Behavioural Adaptation in Human-Robot Interactions

arXiv: https://arxiv.org/abs/2203.10518 · HTML: https://mlsanity.com/paper/2203.10518/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 7 of 7)

- [Minimax Regret Bounds for Reinforcement Learning](https://mlsanity.com/paper/1703.05449/) (2017)
- [Artificial Intelligence for Long-Term Robot Autonomy: A Survey](https://mlsanity.com/paper/1807.05196/) (2018)
- [The STRANDS Project: Long-Term Autonomy in Everyday Environments](https://mlsanity.com/paper/1604.04384/) (2016)
- [Are you still with me? Continuous Engagement Assessment from a Robot's Point of View](https://mlsanity.com/paper/2001.03515/) (2020)
- [Robot gains Social Intelligence through Multimodal Deep Reinforcement Learning](https://mlsanity.com/paper/1702.07492/) (2017)
- [Show, Attend and Interact: Perceivable Human-Robot Social Interaction through Neural Attention Q-Network](https://mlsanity.com/paper/1702.08626/) (2017)
- [Learning to Engage with Interactive Systems: A Field Study on Deep Reinforcement Learning in a Public Museum](https://mlsanity.com/paper/1904.06764/) (2019)

## Built on by (most cited first, 3 of 3)

- [Towards Long-term Autonomy: A Perspective from Robot Learning](https://mlsanity.com/paper/2212.12798/) (2022)
- [User Experience Estimation in Human-Robot Interaction Via Multi-Instance Learning of Multimodal Social Signals](https://mlsanity.com/paper/2507.23544/) (2025)
- [Responsible Personalisation: The Double-Edged Sword of Personalisation in Human-Robot Interaction](https://mlsanity.com/paper/2607.06344/) (2026)

## Abstract (arXiv)

In this work, we propose a framework for allowing autonomous robots deployed for extended periods of time in public spaces to adapt their own behaviour online from user interactions. The robot behaviour planning is embedded in a Reinforcement Learning (RL) framework, where the objective is maximising the level of overall user engagement during the interactions. We use the Upper-Confidence-Bound Value-Iteration (UCBVI) algorithm, which gives a helpful way of managing the exploration-exploitation trade-off for real-time interactions. An engagement model trained end-to-end generates the reward function in real-time during policy execution. We test this approach in a public museum in Lincoln (UK), where the robot is deployed as a tour guide for the visitors. Results show that after a couple of months of exploration, the robot policy learned to maintain the engagement of users for longer, with an increase of 22.8% over the initial static policy in the number of items visited during the tour and a 30% increase in the probability of completing the tour. This work is a promising step toward behavioural adaptation in long-term scenarios for robotics applications in social settings.