# Decoding Emotions: Unveiling Facial Expressions through Acoustic Sensing with Contrastive Attention

arXiv: https://arxiv.org/abs/2410.12811 · HTML: https://mlsanity.com/paper/2410.12811/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 61 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Person Transfer GAN to Bridge Domain Gap for Person Re-Identification](https://mlsanity.com/paper/1711.08565/) (2017)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks](https://mlsanity.com/paper/2105.02358/) (2021)
- [Masked Face Recognition Dataset and Application](https://mlsanity.com/paper/2003.09093/) (2020)
- [The Effect of Wearing a Mask on Face Recognition Performance: an Exploratory Study](https://mlsanity.com/paper/2007.13521/) (2020)
- [WiFE: WiFi and Vision based Intelligent Facial-Gesture Emotion Recognition](https://mlsanity.com/paper/2004.09889/) (2020)

## Abstract (arXiv)

Expression recognition holds great promise for applications such as content recommendation and mental healthcare by accurately detecting users' emotional states. Traditional methods often rely on cameras or wearable sensors, which raise privacy concerns and add extra device burdens. In addition, existing acoustic-based methods struggle to maintain satisfactory performance when there is a distribution shift between the training dataset and the inference dataset. In this paper, we introduce FacER+, an active acoustic facial expression recognition system, which eliminates the requirement for external microphone arrays. FacER+ extracts facial expression features by analyzing the echoes of near-ultrasound signals emitted between the 3D facial contour and the earpiece speaker on a smartphone. This approach not only reduces background noise but also enables the identification of different expressions from various users with minimal training data. We develop a contrastive external attention-based model to consistently learn expression features across different users, reducing the distribution differences. Extensive experiments involving 20 volunteers, both with and without masks, demonstrate that FacER+ can accurately recognize six common facial expressions with over 90% accuracy in diverse, user-independent real-life scenarios, surpassing the performance of the leading acoustic sensing methods by 10%. FacER+ offers a robust and practical solution for facial expression recognition.