# Attribute Inference Attack of Speech Emotion Recognition in Federated Learning Settings

arXiv: https://arxiv.org/abs/2112.13416 · HTML: https://mlsanity.com/paper/2112.13416/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 26 Dec 2021.

## Builds on (most cited first, 21 of 21)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization](https://mlsanity.com/paper/2012.06659/) (2020)
- [Privacy in Deep Learning: A Survey](https://mlsanity.com/paper/2004.12254/) (2020)
- [Vector-Quantized Autoregressive Predictive Coding](https://mlsanity.com/paper/2005.08392/) (2020)
- [Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies](https://mlsanity.com/paper/2011.00406/) (2020)
- [Towards General Deep Leakage in Federated Learning](https://mlsanity.com/paper/2110.09074/) (2021)
- [Securing Secure Aggregation: Mitigating Multi-Round Privacy Leakage in Federated Learning](https://mlsanity.com/paper/2106.03328/) (2021)
- [Source Inference Attacks in Federated Learning](https://mlsanity.com/paper/2109.05659/) (2021)
- [Attention Based Fully Convolutional Network for Speech Emotion Recognition](https://mlsanity.com/paper/1806.01506/) (2018)
- [Privacy Enhanced Multimodal Neural Representations for Emotion Recognition](https://mlsanity.com/paper/1910.13212/) (2019)
- [Byzantine-Robust and Privacy-Preserving Framework for FedML](https://mlsanity.com/paper/2105.02295/) (2021)

## Built on by (most cited first, 12 of 12)

- [FedMultimodal: A Benchmark For Multimodal Federated Learning](https://mlsanity.com/paper/2306.09486/) (2023)
- [A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness](https://mlsanity.com/paper/2212.09006/) (2022)
- [Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labeling](https://mlsanity.com/paper/2203.08810/) (2022)
- [Decoupled Federated Learning for ASR with Non-IID Data](https://mlsanity.com/paper/2206.09102/) (2022)
- [Analyzing Inference Privacy Risks Through Gradients in Machine Learning](https://mlsanity.com/paper/2408.16913/) (2024)
- [RecUP-FL: Reconciling Utility and Privacy in Federated Learning via User-configurable Privacy Defense](https://mlsanity.com/paper/2304.05135/) (2023)
- [Inference Attacks: A Taxonomy, Survey, and Promising Directions](https://mlsanity.com/paper/2406.02027/) (2024)
- [Data Leakage via Access Patterns of Sparse Features in Deep Learning-based Recommendation Systems](https://mlsanity.com/paper/2212.06264/) (2022)
- [SoK: Memorisation in machine learning](https://mlsanity.com/paper/2311.03075/) (2023)
- [Attribute Inference Attacks for Federated Regression Tasks](https://mlsanity.com/paper/2411.12697/) (2024)
- [Partial Federated Learning](https://mlsanity.com/paper/2403.01615/) (2024)
- [Exploring Audio Editing Features as User-Centric Privacy Defenses Against Large Language Model(LLM) Based Emotion Inference Attacks](https://mlsanity.com/paper/2501.18727/) (2025)

## Abstract (arXiv)

Speech emotion recognition (SER) processes speech signals to detect and characterize expressed perceived emotions. Many SER application systems often acquire and transmit speech data collected at the client-side to remote cloud platforms for inference and decision making. However, speech data carry rich information not only about emotions conveyed in vocal expressions, but also other sensitive demographic traits such as gender, age and language background. Consequently, it is desirable for SER systems to have the ability to classify emotion constructs while preventing unintended/improper inferences of sensitive and demographic information. Federated learning (FL) is a distributed machine learning paradigm that coordinates clients to train a model collaboratively without sharing their local data. This training approach appears secure and can improve privacy for SER. However, recent works have demonstrated that FL approaches are still vulnerable to various privacy attacks like reconstruction attacks and membership inference attacks. Although most of these have focused on computer vision applications, such information leakages exist in the SER systems trained using the FL technique. To assess the information leakage of SER systems trained using FL, we propose an attribute inference attack framework that infers sensitive attribute information of the clients from shared gradients or model parameters, corresponding to the FedSGD and the FedAvg training algorithms, respectively. As a use case, we empirically evaluate our approach for predicting the client's gender information using three SER benchmark datasets: IEMOCAP, CREMA-D, and MSP-Improv. We show that the attribute inference attack is achievable for SER systems trained using FL. We further identify that most information leakage possibly comes from the first layer in the SER model.