# Pretrained audio neural networks for Speech emotion recognition in Portuguese

arXiv: https://arxiv.org/abs/2210.14716 · HTML: https://mlsanity.com/paper/2210.14716/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Multimodal Speech Emotion Recognition Using Audio and Text](https://mlsanity.com/paper/1810.04635/) (2018)
- [Automatic tagging using deep convolutional neural networks](https://mlsanity.com/paper/1606.00298/) (2016)
- [Transfer Learning for Speech and Language Processing](https://mlsanity.com/paper/1511.06066/) (2015)
- [Learning Alignment for Multimodal Emotion Recognition from Speech](https://mlsanity.com/paper/1909.05645/) (2019)
- [Speech Emotion Recognition Using Multi-hop Attention Mechanism](https://mlsanity.com/paper/1904.10788/) (2019)
- [Weakly Labelled AudioSet Tagging with Attention Neural Networks](https://mlsanity.com/paper/1903.00765/) (2019)
- [Efficient Speech Emotion Recognition Using Multi-Scale CNN and Attention](https://mlsanity.com/paper/2106.04133/) (2021)
- [Cross-task learning for audio tagging, sound event detection and spatial localization: DCASE 2019 baseline systems](https://mlsanity.com/paper/1904.03476/) (2019)
- [Audio MFCC-gram Transformers for respiratory insufficiency detection in COVID-19](https://mlsanity.com/paper/2210.14085/) (2022)

## Built on by (most cited first, 3 of 3)

- [Acoustic models of Brazilian Portuguese Speech based on Neural Transformers](https://mlsanity.com/paper/2312.09265/) (2023)
- [Discriminant audio properties in deep learning based respiratory insufficiency detection in Brazilian Portuguese](https://mlsanity.com/paper/2405.17569/) (2024)
- [Contrasting Deep Learning Models for Direct Respiratory Insufficiency Detection Versus Blood Oxygen Saturation Estimation](https://mlsanity.com/paper/2407.20989/) (2024)

## Abstract (arXiv)

The goal of speech emotion recognition (SER) is to identify the emotional aspects of speech. The SER challenge for Brazilian Portuguese speech was proposed with short snippets of Portuguese which are classified as neutral, non-neutral female and non-neutral male according to paralinguistic elements (laughing, crying, etc). This dataset contains about 50 minutes of Brazilian Portuguese speech. As the dataset leans on the small side, we investigate whether a combination of transfer learning and data augmentation techniques can produce positive results. Thus, by combining a data augmentation technique called SpecAugment, with the use of Pretrained Audio Neural Networks (PANNs) for transfer learning we are able to obtain interesting results. The PANNs (CNN6, CNN10 and CNN14) are pretrained on a large dataset called AudioSet containing more than 5000 hours of audio. They were finetuned on the SER dataset and the best performing model (CNN10) on the validation set was submitted to the challenge, achieving an F1 score of 0.73 up from 0.54 from the baselines provided by the challenge. Moreover, we also tested the use of Transformer neural architecture, pretrained on about 600 hours of Brazilian Portuguese audio data. Transformers, as well as more complex models of PANNs (CNN14), fail to generalize to the test set in the SER dataset and do not beat the baseline. Considering the limitation of the dataset sizes, currently the best approach for SER is using PANNs (specifically, CNN6 and CNN10).