# Acoustic models of Brazilian Portuguese Speech based on Neural Transformers

arXiv: https://arxiv.org/abs/2312.09265 · HTML: https://mlsanity.com/paper/2312.09265/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech](https://mlsanity.com/paper/1803.08976/) (2018)
- [Very Deep Self-Attention Networks for End-to-End Speech Recognition](https://mlsanity.com/paper/1904.13377/) (2019)
- [Audio Word2Vec: Unsupervised Learning of Audio Segment Representations using Sequence-to-sequence Autoencoder](https://mlsanity.com/paper/1603.00982/) (2016)
- [Self-Attentional Acoustic Models](https://mlsanity.com/paper/1803.09519/) (2018)
- [Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks](https://mlsanity.com/paper/1910.10387/) (2019)
- [Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion](https://mlsanity.com/paper/1905.11563/) (2019)
- [CORAA: a large corpus of spontaneous and prepared speech manually validated for speech recognition in Brazilian Portuguese](https://mlsanity.com/paper/2110.15731/) (2021)
- [Audio MFCC-gram Transformers for respiratory insufficiency detection in COVID-19](https://mlsanity.com/paper/2210.14085/) (2022)
- [Pretrained audio neural networks for Speech emotion recognition in Portuguese](https://mlsanity.com/paper/2210.14716/) (2022)
- [Interpretability Analysis of Deep Models for COVID-19 Detection](https://mlsanity.com/paper/2211.14372/) (2022)

## Built on by (most cited first, 1 of 1)

- [Contrasting Deep Learning Models for Direct Respiratory Insufficiency Detection Versus Blood Oxygen Saturation Estimation](https://mlsanity.com/paper/2407.20989/) (2024)

## Abstract (arXiv)

An acoustic model, trained on a significant amount of unlabeled data, consists of a self-supervised learned speech representation useful for solving downstream tasks, perhaps after a fine-tuning of the model in the respective downstream task. In this work, we build an acoustic model of Brazilian Portuguese Speech through a Transformer neural network. This model was pretrained on more than 800 hours of Brazilian Portuguese Speech, using a combination of pretraining techniques. Using a labeled dataset collected for the detection of respiratory insufficiency in Brazilian Portuguese speakers, we fine-tune the pretrained Transformer neural network on the following tasks: respiratory insufficiency detection, gender recognition and age group classification. We compare the performance of pretrained Transformers on these tasks with that of Transformers without previous pretraining, noting a significant improvement. In particular, the performance of respiratory insufficiency detection obtains the best reported results so far, indicating this kind of acoustic model as a promising tool for speech-as-biomarker approach. Moreover, the performance of gender recognition is comparable to the state of the art models in English.