# A trained humanoid robot can perform human-like crossmodal social attention and conflict resolution

arXiv: https://arxiv.org/abs/2111.01906 · HTML: https://mlsanity.com/paper/2111.01906/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 98 references that are in the map.
- First posted on arXiv 2 Nov 2021.

## Builds on (most cited first, 13 of 13)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016)
- [AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection](https://mlsanity.com/paper/1901.01342/) (2019)
- [2.5D Visual Sound](https://mlsanity.com/paper/1812.04204/) (2018)
- [ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction](https://mlsanity.com/paper/2012.06170/) (2020)
- [STAViS: Spatio-Temporal AudioVisual Saliency Network](https://mlsanity.com/paper/2001.03063/) (2020)
- [How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild](https://mlsanity.com/paper/2106.03932/) (2021)
- [FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection](https://mlsanity.com/paper/2109.00577/) (2021)
- [A Neurorobotic Experiment for Crossmodal Conflict Resolution in Complex Environments](https://mlsanity.com/paper/1802.10408/) (2018)
- [Assessing the Contribution of Semantic Congruency to Multisensory Integration and Conflict Resolution](https://mlsanity.com/paper/1810.06748/) (2018)

## Built on by (most cited first, 1 of 1)

- [Unified Dynamic Scanpath Predictors Outperform Individually Trained Neural Models](https://mlsanity.com/paper/2405.02929/) (2024)

## Abstract (arXiv)

To enhance human-robot social interaction, it is essential for robots to process multiple social cues in a complex real-world environment. However, incongruency of input information across modalities is inevitable and could be challenging for robots to process. To tackle this challenge, our study adopted the neurorobotic paradigm of crossmodal conflict resolution to make a robot express human-like social attention. A behavioural experiment was conducted on 37 participants for the human study. We designed a round-table meeting scenario with three animated avatars to improve ecological validity. Each avatar wore a medical mask to obscure the facial cues of the nose, mouth, and jaw. The central avatar shifted its eye gaze while the peripheral avatars generated sound. Gaze direction and sound locations were either spatially congruent or incongruent. We observed that the central avatar's dynamic gaze could trigger crossmodal social attention responses. In particular, human performances are better under the congruent audio-visual condition than the incongruent condition. Our saliency prediction model was trained to detect social cues, predict audio-visual saliency, and attend selectively for the robot study. After mounting the trained model on the iCub, the robot was exposed to laboratory conditions similar to the human experiment. While the human performances were overall superior, our trained model demonstrated that it could replicate attention responses similar to humans.