# MultiMediate'23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions

arXiv: https://arxiv.org/abs/2308.08256 · HTML: https://mlsanity.com/paper/2308.08256/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 62 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 15 of 15)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [SoundNet: Learning Sound Representations from Unlabeled Video](https://mlsanity.com/paper/1610.09001/) (2016)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge](https://mlsanity.com/paper/1605.01600/) (2016)
- [iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis](https://mlsanity.com/paper/2107.00285/) (2021)
- [Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour](https://mlsanity.com/paper/1801.06055/) (2018)
- [Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation](https://mlsanity.com/paper/2207.12817/) (2022)
- [MultiMediate '22: Backchannel Detection and Agreement Estimation in Group Interactions](https://mlsanity.com/paper/2209.09578/) (2022)
- [Towards Social & Engaging Peer Learning: Predicting Backchanneling and Disengagement in Children](https://mlsanity.com/paper/2007.11346/) (2020)
- [Backchannel Detection and Agreement Estimation from Video with Transformer Networks](https://mlsanity.com/paper/2306.01656/) (2023)
- [Emergent Leadership Detection Across Datasets](https://mlsanity.com/paper/1905.02058/) (2019)
- [Modeling Engagement in Long-Term, In-Home Socially Assistive Robot Interventions for Children with Autism Spectrum Disorders](https://mlsanity.com/paper/2002.02453/) (2020)
- [Can I Trust You? A User Study of Robot Mediation of a Support Group](https://mlsanity.com/paper/2002.04671/) (2020)
- [Developing an Effective and Automated Patient Engagement Estimator for Telehealth: A Machine Learning Approach](https://mlsanity.com/paper/2011.08690/) (2020)

## Built on by (most cited first, 3 of 3)

- [Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on Engagement](https://mlsanity.com/paper/2409.13726/) (2024)
- [Beyond Words: Multimodal LLM Knows When to Speak](https://mlsanity.com/paper/2505.14654/) (2025)
- [MAGIC-TBR: Multiview Attention Fusion for Transformer-based Bodily Behavior Recognition in Group Settings](https://mlsanity.com/paper/2309.10765/) (2023)

## Abstract (arXiv)

Automatic analysis of human behaviour is a fundamental prerequisite for the creation of machines that can effectively interact with- and support humans in social interactions. In MultiMediate'23, we address two key human social behaviour analysis tasks for the first time in a controlled challenge: engagement estimation and bodily behaviour recognition in social interactions. This paper describes the MultiMediate'23 challenge and presents novel sets of annotations for both tasks. For engagement estimation we collected novel annotations on the NOvice eXpert Interaction (NOXI) database. For bodily behaviour recognition, we annotated test recordings of the MPIIGroupInteraction corpus with the BBSI annotation scheme. In addition, we present baseline results for both challenge tasks.