# Multipar-T: Multiparty-Transformer for Capturing Contingent Behaviors in Group Conversations

arXiv: https://arxiv.org/abs/2304.12204 · HTML: https://mlsanity.com/paper/2304.12204/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 71 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Dyadformer: A Multi-modal Transformer for Long-Range Modeling of Dyadic Interactions](https://mlsanity.com/paper/2109.09487/) (2021)
- [Graph-Based Social Relation Reasoning](https://mlsanity.com/paper/2007.07453/) (2020)
- [Are you still with me? Continuous Engagement Assessment from a Robot's Point of View](https://mlsanity.com/paper/2001.03515/) (2020)
- [Class-attention Video Transformer for Engagement Intensity Prediction](https://mlsanity.com/paper/2208.07216/) (2022)
- [Bootstrap Model Ensemble and Rank Loss for Engagement Intensity Regression](https://mlsanity.com/paper/1907.03422/) (2019)
- [Joint Engagement Classification using Video Augmentation Techniques for Multi-person Human-robot Interaction](https://mlsanity.com/paper/2212.14128/) (2022)

## Built on by (most cited first, 1 of 1)

- [HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer](https://mlsanity.com/paper/2305.12369/) (2023)

## Abstract (arXiv)

As we move closer to real-world AI systems, AI agents must be able to deal with multiparty (group) conversations. Recognizing and interpreting multiparty behaviors is challenging, as the system must recognize individual behavioral cues, deal with the complexity of multiple streams of data from multiple people, and recognize the subtle contingent social exchanges that take place amongst group members. To tackle this challenge, we propose the Multiparty-Transformer (Multipar-T), a transformer model for multiparty behavior modeling. The core component of our proposed approach is the Crossperson Attention, which is specifically designed to detect contingent behavior between pairs of people. We verify the effectiveness of Multipar-T on a publicly available video-based group engagement detection benchmark, where it outperforms state-of-the-art approaches in average F-1 scores by 5.2% and individual class F-1 scores by up to 10.0%. Through qualitative analysis, we show that our Crossperson Attention module is able to discover contingent behavior.