# Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning

arXiv: https://arxiv.org/abs/2202.02611 · HTML: https://mlsanity.com/paper/2202.02611/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 5 Feb 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Applied Federated Learning: Improving Google Keyboard Query Suggestions](https://mlsanity.com/paper/1812.02903/) (2018)
- [Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning](https://mlsanity.com/paper/2006.12097/) (2020)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [Training Keyword Spotting Models on Non-IID Data with Federated Learning](https://mlsanity.com/paper/2005.10406/) (2020)
- [Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospective](https://mlsanity.com/paper/2002.11545/) (2020)
- [Attention Based Fully Convolutional Network for Speech Emotion Recognition](https://mlsanity.com/paper/1806.01506/) (2018)
- [Attentive Convolutional Neural Network based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech](https://mlsanity.com/paper/1706.00612/) (2017)
- [Self-supervised audio representation learning for mobile devices](https://mlsanity.com/paper/1905.11796/) (2019)
- [Improved Speech Emotion Recognition using Transfer Learning and Spectrogram Augmentation](https://mlsanity.com/paper/2108.02510/) (2021)
- [CopyPaste: An Augmentation Method for Speech Emotion Recognition](https://mlsanity.com/paper/2010.14602/) (2020)
- [Weak-Attention Suppression For Transformer Based Speech Recognition](https://mlsanity.com/paper/2005.09137/) (2020)
- [Federated Self-Training for Semi-Supervised Audio Recognition](https://mlsanity.com/paper/2107.06877/) (2021)

## Built on by (most cited first, 6 of 6)

- [FedMultimodal: A Benchmark For Multimodal Federated Learning](https://mlsanity.com/paper/2306.09486/) (2023)
- [A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness](https://mlsanity.com/paper/2212.09006/) (2022)
- [Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labeling](https://mlsanity.com/paper/2203.08810/) (2022)
- [GAI-Enabled Explainable Personalized Federated Semi-Supervised Learning](https://mlsanity.com/paper/2410.08634/) (2024)
- [Unified Alignment Protocol: Making Sense of the Unlabeled Data in New Domains](https://mlsanity.com/paper/2505.21010/) (2025)
- [Transformer Redesign for Late Fusion of Audio-Text Features on Ultra-Low-Power Edge Hardware](https://mlsanity.com/paper/2510.18036/) (2025)

## Abstract (arXiv)

Speech Emotion Recognition (SER) refers to the recognition of human emotions from natural speech. If done accurately, it can offer a number of benefits in building human-centered context-aware intelligent systems. Existing SER approaches are largely centralized, without considering users' privacy. Federated Learning (FL) is a distributed machine learning paradigm dealing with decentralization of privacy-sensitive personal data. In this paper, we present a privacy-preserving and data-efficient SER approach by utilizing the concept of FL. To the best of our knowledge, this is the first federated SER approach, which utilizes self-training learning in conjunction with federated learning to exploit both labeled and unlabeled on-device data. Our experimental evaluations on the IEMOCAP dataset shows that our federated approach can learn generalizable SER models even under low availability of data labels and highly non-i.i.d. distributions. We show that our approach with as few as 10% labeled data, on average, can improve the recognition rate by 8.67% compared to the fully-supervised federated counterparts.