# Voice Quality and Pitch Features in Transformer-Based Speech Recognition

arXiv: https://arxiv.org/abs/2112.11391 · HTML: https://mlsanity.com/paper/2112.11391/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Transformers with convolutional context for ASR](https://mlsanity.com/paper/1904.11660/) (2019)
- [Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions](https://mlsanity.com/paper/1904.02619/) (2019)
- [Residual Convolutional CTC Networks for Automatic Speech Recognition](https://mlsanity.com/paper/1702.07793/) (2017)
- [Fully Convolutional Speech Recognition](https://mlsanity.com/paper/1812.06864/) (2018)
- [Scaling Up Online Speech Recognition Using ConvNets](https://mlsanity.com/paper/2001.09727/) (2020)
- [Letter-Based Speech Recognition with Gated ConvNets](https://mlsanity.com/paper/1712.09444/) (2017)
- [Prosodic-Enhanced Siamese Convolutional Neural Networks for Cross-Device Text-Independent Speaker Verification](https://mlsanity.com/paper/1808.01026/) (2018)
- [Convolutional Speech Recognition with Pitch and Voice Quality Features](https://mlsanity.com/paper/2009.01309/) (2020)

## Abstract (arXiv)

Jitter and shimmer measurements have shown to be carriers of voice quality and prosodic information which enhance the performance of tasks like speaker recognition, diarization or automatic speech recognition (ASR). However, such features have been seldom used in the context of neural-based ASR, where spectral features often prevail. In this work, we study the effects of incorporating voice quality and pitch features altogether and separately to a Transformer-based ASR model, with the intuition that the attention mechanisms might exploit latent prosodic traits. For doing so, we propose separated convolutional front-ends for prosodic and spectral features, showing that this architectural choice yields better results than simple concatenation of such pitch and voice quality features to mel-spectrogram filterbanks. Furthermore, we find mean Word Error Rate relative reductions of up to 5.6% with the LibriSpeech benchmark. Such findings motivate further research on the application of prosody knowledge for increasing the robustness of Transformer-based ASR.