# Emotion Recognition in Sign Language Conversation

arXiv: https://arxiv.org/abs/2605.23328 · HTML: https://mlsanity.com/paper/2605.23328/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 34 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 14 of 14)

- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose](https://mlsanity.com/paper/2303.07399/) (2023)
- [COSMIC: COmmonSense knowledge for eMotion Identification in Conversations](https://mlsanity.com/paper/2010.02795/) (2020)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [Quantitative Survey of the State of the Art in Sign Language Recognition](https://mlsanity.com/paper/2008.09918/) (2020)
- [BBC-Oxford British Sign Language Dataset](https://mlsanity.com/paper/2111.03635/) (2021)
- [Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition](https://mlsanity.com/paper/2311.11009/) (2023)
- [Deep Emotion Recognition in Textual Conversations: A Survey](https://mlsanity.com/paper/2211.09172/) (2022)
- [TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2401.12987/) (2024)
- [STUDIES: Corpus of Japanese Empathetic Dialogue Speech Towards Friendly Voice Agent](https://mlsanity.com/paper/2203.14757/) (2022)
- [Affective Video Content Analysis: Decade Review and New Perspectives](https://mlsanity.com/paper/2310.17212/) (2023)
- [EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language](https://mlsanity.com/paper/2505.17090/) (2025)
- [Emotion Recognition in Signers](https://mlsanity.com/paper/2512.15376/) (2025)

## Abstract (arXiv)

Emotion Recognition in Conversation is a core component of affective computing, while current sign language emotion datasets primarily focus on isolated sentences and lack conversational context. Models trained exclusively on these isolated utterances demonstrate degraded performance in real world scenarios because they cannot utilize historical dialogue flow. To address this structural limitation, we introduce the ERC task to sign language video analysis and propose the eJSL Dialog dataset. Constructed using the scripts from the STUDIES corpus, the dataset contains 1,920 video samples organized into 480 unique dialogues. We conduct systematic benchmarking on this dataset using models ranging from isolated visual networks to multimodal conversational architectures. The results suggest the feasibility of extending conversational ERC frameworks to sign-language dialogue under the current benchmark setting, while also revealing limitations in existing visual representations for capturing sign-specific affective cues, motivating future work on sign-specific visual modeling and larger sign-language conversational training resources.