# Universal Paralinguistic Speech Representations Using Self-Supervised Conformers

arXiv: https://arxiv.org/abs/2110.04621 · HTML: https://mlsanity.com/paper/2110.04621/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 9 Oct 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis](https://mlsanity.com/paper/1806.04558/) (2018)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection](https://mlsanity.com/paper/1904.05441/) (2019)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks](https://mlsanity.com/paper/1904.03416/) (2019)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [Personalizing ASR for Dysarthric and Accented Speech with Limited Data](https://mlsanity.com/paper/1907.13511/) (2019)
- [Unsupervised Learning of Semantic Audio Representations](https://mlsanity.com/paper/1711.02209/) (2017)
- [Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision](https://mlsanity.com/paper/1911.05894/) (2019)
- [FRILL: A Non-Semantic Speech Embedding for Mobile Devices](https://mlsanity.com/paper/2011.04609/) (2020)
- [Spoken Language Identification using ConvNets](https://mlsanity.com/paper/1910.04269/) (2019)
- [Comparing Supervised Models And Learned Speech Representations For Classifying Intelligibility Of Disordered Speech On Selected Phrases](https://mlsanity.com/paper/2107.03985/) (2021)

## Built on by (most cited first, 30 of 32)

- [Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning](https://mlsanity.com/paper/2201.03529/) (2022)
- [A Large-Scale Evaluation of Speech Foundation Models](https://mlsanity.com/paper/2404.09385/) (2024)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks](https://mlsanity.com/paper/2303.00733/) (2023)
- [TRILLsson: Distilled Universal Paralinguistic Speech Representations](https://mlsanity.com/paper/2203.00236/) (2022)
- [HeAR -- Health Acoustic Representations](https://mlsanity.com/paper/2403.02522/) (2024)
- [BYOL-S: Learning Self-supervised Speech Representations by Bootstrapping](https://mlsanity.com/paper/2206.12038/) (2022)
- [Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model](https://mlsanity.com/paper/2304.11976/) (2023)
- [UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions](https://mlsanity.com/paper/2310.02973/) (2023)
- [Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settings](https://mlsanity.com/paper/2303.05737/) (2023)
- [StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models](https://mlsanity.com/paper/2311.16509/) (2023)
- [Hybrid Handcrafted and Learnable Audio Representation for Analysis of Speech Under Cognitive and Physical Load](https://mlsanity.com/paper/2203.16637/) (2022)
- [Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP](https://mlsanity.com/paper/2304.00860/) (2023)
- [Ultra-Low-Bitrate Speech Coding with Pretrained Transformers](https://mlsanity.com/paper/2207.02262/) (2022)
- [LanSER: Language-Model Supported Speech Emotion Recognition](https://mlsanity.com/paper/2309.03978/) (2023)
- [Automatic Data Augmentation Selection and Parametrization in Contrastive Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2204.04170/) (2022)
- [Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representations](https://mlsanity.com/paper/2305.08099/) (2023)
- [The Need for Medically Aware Video Compression in Gastroenterology](https://mlsanity.com/paper/2211.01472/) (2022)
- [Learning the joint distribution of two sequences using little or no paired data](https://mlsanity.com/paper/2212.03232/) (2022)
- [Improved Cross-Lingual Transfer Learning For Automatic Speech Translation](https://mlsanity.com/paper/2306.00789/) (2023)
- [Introspective Action Advising for Interpretable Transfer Learning](https://mlsanity.com/paper/2306.12314/) (2023)
- [Optimizing Audio Augmentations for Contrastive Learning of Health-Related Acoustic Signals](https://mlsanity.com/paper/2309.05843/) (2023)
- [Multitask vocal burst modeling with ResNets and pre-trained paralinguistic Conformers](https://mlsanity.com/paper/2206.12494/) (2022)
- [Knowledge distillation for fast and accurate DNA sequence correction](https://mlsanity.com/paper/2211.09862/) (2022)
- [Are Paralinguistic Representations all that is needed for Speech Emotion Recognition?](https://mlsanity.com/paper/2402.01579/) (2024)
- [Voice Quality Dimensions as Interpretable Primitives for Speaking Style for Atypical Speech and Affect](https://mlsanity.com/paper/2505.21809/) (2025)
- [Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices](https://mlsanity.com/paper/2207.05784/) (2022)
- [The unreasonable effectiveness of AI CADe polyp detectors to generalize to new countries](https://mlsanity.com/paper/2312.06833/) (2023)
- [Predicting Generalization of AI Colonoscopy Models to Unseen Data](https://mlsanity.com/paper/2403.09920/) (2024)
- [Towards Cross-Lingual Audio Abuse Detection in Low-Resource Settings with Few-Shot Learning](https://mlsanity.com/paper/2412.01408/) (2024)

## Abstract (arXiv)

Many speech applications require understanding aspects beyond the words being spoken, such as recognizing emotion, detecting whether the speaker is wearing a mask, or distinguishing real from synthetic speech. In this work, we introduce a new state-of-the-art paralinguistic representation derived from large-scale, fully self-supervised training of a 600M+ parameter Conformer-based architecture. We benchmark on a diverse set of speech tasks and demonstrate that simple linear classifiers trained on top of our time-averaged representation outperform nearly all previous results, in some cases by large margins. Our analyses of context-window size demonstrate that, surprisingly, 2 second context-windows achieve 96% the performance of the Conformers that use the full long-term context on 7 out of 9 tasks. Furthermore, while the best per-task representations are extracted internally in the network, stable performance across several layers allows a single universal representation to reach near optimal performance on all tasks.