mlsanityData to 7 Oct 2026

Multi Modal Facial Expression Recognition with Transformer-Based Fusion Networks and Dynamic Sampling

arXiv 2303.08419 ↗First posted 15 Mar 2023cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Facial expression recognition is an essential task for various applications, including emotion detection, mental health analysis, and human-machine interactions. In this paper, we propose a multi-modal facial expression recognition method that exploits audio information along with facial images to provide a crucial clue to differentiate some ambiguous facial expressions. Specifically, we introduce a Modal Fusion Module (MFM) to fuse audio-visual information, where image and audio features are extracted from Swin Transformer. Additionally, we tackle the imbalance problem in the dataset by employing dynamic data resampling. Our model has been evaluated in the Affective Behavior in-the-wild (ABAW) challenge of CVPR 2023.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 14 of its 20 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

14 of 20 references.

References · 14

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2019Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFacecited by 222
  4. 2018Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyondcited by 216
  5. 2020Analysing Affective Behavior in the First ABAW 2020 Competitioncited by 178
  6. 2021Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Frameworkcited by 174
  7. 2021Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Studycited by 168
  8. 2021Analysing Affective Behavior in the second ABAW2 Competitioncited by 163
  9. 2022ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challengescited by 156
  10. 2019Face Behavior a la carte: Expressions, Affect and Action Units in a Single Networkcited by 155
  11. 2022ABAW: Learning from Synthetic Data & Multi-Task Learning Challengescited by 110
  12. 2023ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challengescited by 110
2 more
  1. 2022Transformer-based Multimodal Information Fusion for Facial Expression Analysiscited by 49
  2. 2018Estimate and Replace: A Novel Approach to Integrating Deep Neural Networks with Existing Applicationscited by 21

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024The 6th Affective Behavior Analysis in-the-wild (ABAW) Competitioncited by 78
  2. 20247th ABAW Competition: Multi-Task Learning and Compound Expression Recognitioncited by 53