mlsanityData to 7 Oct 2026

Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition

arXiv 2303.08356 ↗First posted 15 Mar 2023cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Human emotion recognition plays an important role in human-computer interaction. In this paper, we present our approach to the Valence-Arousal (VA) Estimation Challenge, Expression (Expr) Classification Challenge, and Action Unit (AU) Detection Challenge of the 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). Specifically, we propose a novel multi-modal fusion model that leverages Temporal Convolutional Networks (TCN) and Transformer to enhance the performance of continuous emotion recognition. Our model aims to effectively integrate visual and audio information for improved accuracy in recognizing emotions. Our model outperforms the baseline and ranks 3 in the Expression Classification challenge.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

24 of 33 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  3. 2019Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFacecited by 222
  4. 2018Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyondcited by 216
  5. 2016Temporal Convolutional Networks: A Unified Approach to Action Segmentationcited by 199
  6. 2020Analysing Affective Behavior in the First ABAW 2020 Competitioncited by 178
  7. 2021Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Frameworkcited by 174
  8. 2021Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Studycited by 168
  9. 2021Analysing Affective Behavior in the second ABAW2 Competitioncited by 163
  10. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 158
  11. 2022ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challengescited by 156
  12. 2019Face Behavior a la carte: Expressions, Affect and Action Units in a Single Networkcited by 155
12 more
  1. 2022ABAW: Learning from Synthetic Data & Multi-Task Learning Challengescited by 110
  2. 2023ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challengescited by 110
  3. 2022Killing Two Birds with One Stone:Efficient and Robust Training of Face Recognition CNNs by Partial FCcited by 73
  4. 2017End-to-End Multimodal Emotion Recognition using Deep Neural Networkscited by 69
  5. 2021Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognitioncited by 55
  6. 2020Two-Stream Aural-Visual Affect Analysis in the Wildcited by 51
  7. 2021A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognitioncited by 24
  8. 2020Detecting expressions with multimodal transformerscited by 14
  9. 2021Multitask Multi-database Emotion Recognitioncited by 12
  10. 2022Parallel Spatio-Temporal Attention-Based TCN for Multivariate Time Series Predictioncited by 12
  11. 2020Deep Auto-Encoders with Sequential Learning for Multimodal Dimensional Emotion Recognitioncited by 11
  12. 2019Emotion Recognition Using Fusion of Audio and Video Featurescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 5; 2026: 4.

20 papers.

Sort:

  1. 2024Robust Light-Weight Facial Affective Behavior Recognition with CLIPcited by 16
  2. 2024Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkitcited by 16
  3. 2024Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognitioncited by 15
  4. 2025HSEmotion Team at ABAW-8 Competition: Audiovisual Ambivalence/Hesitancy, Emotional Mimicry Intensity and Facial Expression Recognitioncited by 15
  5. 2024Joint Multimodal Transformer for Emotion Recognition in the Wildcited by 13
  6. 2024Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wildcited by 9
  7. 2024HSEmotion Team at the 6th ABAW Competition: Facial Expressions, Valence-Arousal and Emotion Intensity Predictioncited by 8
  8. 2024Rethinking Affect Analysis: A Protocol for Ensuring Fairness and Consistencycited by 7
  9. 2024Emotion Recognition Using Transformers with Masked Learningcited by 4
  10. 2024AUD-TGN: Advancing Action Unit Detection with Temporal Convolution and GPT-2 in Wild Audiovisual Contextscited by 3
  11. 2024Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognitioncited by 3
  12. 2025CausalAffect: Causal Discovery for Facial Affective Understandingcited by 3
8 more
  1. 2024Inconsistency-Aware Cross-Attention for Audio-Visual Fusion in Dimensional Emotion Recognitioncited by 1
  2. 2025Interpretable Concept-based Deep Learning Framework for Multimodal Human Behavior Modelingcited by 1
  3. 2026Stage-Adaptive Reliability Modeling for Continuous Valence-Arousal Estimationcited by 1
  4. 2026Multimodal Emotion Recognition via Bi-directional Cross-Attention and Temporal Modelingcited by 1
  5. 2026HSEmotion Team at ABAW-10 Competition: Facial Expression Recognition, Valence-Arousal Estimation, Action Unit Detection and Fine-Grained Violence Classificationcited by 1
  6. 2026Distance-aware Soft Prompt Guidance for Multimodal Valence-Arousal Estimationcited by 1
  7. 2025Emotion estimation from video footage with LSTMcited by 0
  8. 2025United we stand, Divided we fall: Handling Weak Complementary Relationships for Audio-Visual Emotion Recognition in Valence-Arousal Spacecited by 0