# Multi-scale multi-modal micro-expression recognition algorithm based on transformer

arXiv: https://arxiv.org/abs/2301.02969 · HTML: https://mlsanity.com/paper/2301.02969/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 50 references that are in the map.
- First posted on arXiv 8 Jan 2023.

## Builds on (most cited first, 14 of 14)

- [CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2203.00680/) (2022)
- [TransFG: A Transformer Architecture for Fine-grained Recognition](https://mlsanity.com/paper/2103.07976/) (2021)
- [OFF-ApexNet on Micro-expression Recognition System](https://mlsanity.com/paper/1805.08699/) (2018)
- [Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition](https://mlsanity.com/paper/1902.03634/) (2019)
- [RAMS-Trans: Recurrent Attention Multi-scale Transformer forFine-grained Image Recognition](https://mlsanity.com/paper/2107.08192/) (2021)
- [Less is More: Micro-expression Recognition from Video using Apex Frame](https://mlsanity.com/paper/1606.01721/) (2016)
- [Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-expressions](https://mlsanity.com/paper/1901.04656/) (2019)
- [LEARNet Dynamic Imaging Network for Micro Expression Recognition](https://mlsanity.com/paper/1904.09410/) (2019)
- [Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition](https://mlsanity.com/paper/2112.05851/) (2021)
- [MMNet: Muscle motion-guided network for micro-expression recognition](https://mlsanity.com/paper/2201.05297/) (2022)
- [Micro-Attention for Micro-Expression recognition](https://mlsanity.com/paper/1811.02360/) (2018)
- [AffectiveNet: Affective-Motion Feature Learningfor Micro Expression Recognition](https://mlsanity.com/paper/2104.07569/) (2021)
- [Spontaneous Subtle Expression Detection and Recognition based on Facial Strain](https://mlsanity.com/paper/1606.02792/) (2016)
- [Seeking Salient Facial Regions for Cross-Database Micro-Expression Recognition](https://mlsanity.com/paper/2111.15361/) (2021)

## Built on by (most cited first, 1 of 1)

- [Micro-expression Recognition Based on Dual-branch Feature Extraction and Fusion](https://mlsanity.com/paper/2602.23950/) (2026)

## Abstract (arXiv)

A micro-expression is a spontaneous unconscious facial muscle movement that can reveal the true emotions people attempt to hide. Although manual methods have made good progress and deep learning is gaining prominence. Due to the short duration of micro-expression and different scales of expressed in facial regions, existing algorithms cannot extract multi-modal multi-scale facial region features while taking into account contextual information to learn underlying features. Therefore, in order to solve the above problems, a multi-modal multi-scale algorithm based on transformer network is proposed in this paper, aiming to fully learn local multi-grained features of micro-expressions through two modal features of micro-expressions - motion features and texture features. To obtain local area features of the face at different scales, we learned patch features at different scales for both modalities, and then fused multi-layer multi-headed attention weights to obtain effective features by weighting the patch features, and combined cross-modal contrastive learning for model optimization. We conducted comprehensive experiments on three spontaneous datasets, and the results show the accuracy of the proposed algorithm in single measurement SMIC database is up to 78.73% and the F1 value on CASMEII of the combined database is up to 0.9071, which is at the leading level.