# Robot gains Social Intelligence through Multimodal Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1702.07492 · HTML: https://mlsanity.com/paper/1702.07492/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2017.

## Built on by (most cited first, 7 of 7)

- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [A Survey on Human-aware Robot Navigation](https://mlsanity.com/paper/2106.11650/) (2021)
- [The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMs](https://mlsanity.com/paper/2602.11583/) (2026)
- [Learning on the Job: Long-Term Behavioural Adaptation in Human-Robot Interactions](https://mlsanity.com/paper/2203.10518/) (2022)
- [Autonomous Human-Robot Interaction via Operator Imitation](https://mlsanity.com/paper/2504.02724/) (2025)
- [A General, Evolution-Inspired Reward Function for Social Robotics](https://mlsanity.com/paper/2202.00617/) (2022)
- [Learning to Communicate Functional States with Nonverbal Expressions for Improved Human-Robot Collaboration](https://mlsanity.com/paper/2404.19253/) (2024)

## Abstract (arXiv)

For robots to coexist with humans in a social world like ours, it is crucial that they possess human-like social interaction skills. Programming a robot to possess such skills is a challenging task. In this paper, we propose a Multimodal Deep Q-Network (MDQN) to enable a robot to learn human-like interaction skills through a trial and error method. This paper aims to develop a robot that gathers data during its interaction with a human and learns human interaction behaviour from the high-dimensional sensory information using end-to-end reinforcement learning. This paper demonstrates that the robot was able to learn basic interaction skills successfully, after 14 days of interacting with people.