# Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognition

arXiv: https://arxiv.org/abs/2412.13541 · HTML: https://mlsanity.com/paper/2412.13541/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 112 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 30 of 34)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Look at Boundary: A Boundary-Aware Face Alignment Algorithm](https://mlsanity.com/paper/1805.10483/) (2018)
- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [Distortion Agnostic Deep Watermarking](https://mlsanity.com/paper/2001.04580/) (2020)
- [UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2211.11256/) (2022)
- [MARLIN: Masked Autoencoder for facial video Representation LearnINg](https://mlsanity.com/paper/2211.06627/) (2022)
- [Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2312.06337/) (2023)
- [COGMEN: COntextualized GNN based Multimodal Emotion recognitioN](https://mlsanity.com/paper/2205.02455/) (2022)
- [Rethinking Graph Neural Architecture Search from Message-passing](https://mlsanity.com/paper/2103.14282/) (2021)
- [Pre-training strategies and datasets for facial representation learning](https://mlsanity.com/paper/2103.16554/) (2021)
- [FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition](https://mlsanity.com/paper/1609.06591/) (2016)
- [Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition](https://mlsanity.com/paper/2201.05834/) (2022)
- [MER-GCN: Micro Expression Recognition Based on Relation Modeling with Graph Convolutional Network](https://mlsanity.com/paper/2004.08915/) (2020)
- [GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion Recognition](https://mlsanity.com/paper/2207.12261/) (2022)
- [Subtype-aware Unsupervised Domain Adaptation for Medical Diagnosis](https://mlsanity.com/paper/2101.00318/) (2021)
- [Variational Metric Scaling for Metric-Based Meta-Learning](https://mlsanity.com/paper/1912.11809/) (2019)
- [SepTr: Separable Transformer for Audio Spectrogram Processing](https://mlsanity.com/paper/2203.09581/) (2022)
- [Cluster-Level Contrastive Learning for Emotion Recognition in Conversations](https://mlsanity.com/paper/2302.03508/) (2023)
- [Towards Task Sampler Learning for Meta-Learning](https://mlsanity.com/paper/2307.08924/) (2023)
- [PropagationNet: Propagate Points to Curve to Learn Structure Information](https://mlsanity.com/paper/2006.14308/) (2020)
- [Hacking Task Confounder in Meta-Learning](https://mlsanity.com/paper/2312.05771/) (2023)
- [A Real Time Facial Expression Classification System Using Local Binary Patterns](https://mlsanity.com/paper/1505.04058/) (2015)
- [Facial expression recognition based on local region specific features and support vector machines](https://mlsanity.com/paper/1604.04337/) (2016)

## Abstract (arXiv)

Fine-grained emotion recognition (FER) plays a vital role in various fields, such as disease diagnosis, personalized recommendations, and multimedia mining. However, existing FER methods face three key challenges in real-world applications: (i) they rely on large amounts of continuously annotated data to ensure accuracy since emotions are complex and ambiguous in reality, which is costly and time-consuming; (ii) they cannot capture the temporal heterogeneity caused by changing emotion patterns, because they usually assume that the temporal correlation within sampling periods is the same; (iii) they do not consider the spatial heterogeneity of different FER scenarios, that is, the distribution of emotion information in different data may have bias or interference. To address these challenges, we propose a Spatio-Temporal Fuzzy-oriented Multi-modal Meta-learning framework (ST-F2M). Specifically, ST-F2M first divides the multi-modal videos into multiple views, and each view corresponds to one modality of one emotion. Multiple randomly selected views for the same emotion form a meta-training task. Next, ST-F2M uses an integrated module with spatial and temporal convolutions to encode the data of each task, reflecting the spatial and temporal heterogeneity. Then it adds fuzzy semantic information to each task based on generalized fuzzy rules, which helps handle the complexity and ambiguity of emotions. Finally, ST-F2M learns emotion-related general meta-knowledge through meta-recurrent neural networks to achieve fast and robust fine-grained emotion recognition. Extensive experiments show that ST-F2M outperforms various state-of-the-art methods in terms of accuracy and model efficiency. In addition, we construct ablation studies and further analysis to explore why ST-F2M performs well.