mlsanityData to 7 Oct 2026

Universal Paralinguistic Speech Representations Using Self-Supervised Conformers

arXiv 2110.04621 ↗First posted 9 Oct 2021cs.SD cs.LG eess.AScited by 32

Not on the map. Why?

Abstract (arXiv)

Many speech applications require understanding aspects beyond the words being spoken, such as recognizing emotion, detecting whether the speaker is wearing a mask, or distinguishing real from synthetic speech. In this work, we introduce a new state-of-the-art paralinguistic representation derived from large-scale, fully self-supervised training of a 600M+ parameter Conformer-based architecture. We benchmark on a diverse set of speech tasks and demonstrate that simple linear classifiers trained on top of our time-averaged representation outperform nearly all previous results, in some cases by large margins. Our analyses of context-window size demonstrate that, surprisingly, 2 second context-windows achieve 96% the performance of the Conformers that use the full long-term context on 7 out of 9 tasks. Furthermore, while the best per-task representations are extracted internally in the network, stable performance across several layers allows a single universal representation to reach near optimal performance on all tasks.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 9 Oct 2021.

Builds on

30 of 42 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2019Searching for MobileNetV3cited by 1,984
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  9. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  10. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  11. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  12. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
18 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  2. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  3. 2017Look, Listen and Learncited by 452
  4. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  5. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  6. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  7. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  8. 2019ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detectioncited by 173
  9. 2020Contrastive Learning of General-Purpose Audio Representationscited by 168
  10. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  11. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  12. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  13. 2019Personalizing ASR for Dysarthric and Accented Speech with Limited Datacited by 45
  14. 2017Unsupervised Learning of Semantic Audio Representationscited by 44
  15. 2019Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervisioncited by 27
  16. 2020FRILL: A Non-Semantic Speech Embedding for Mobile Devicescited by 13
  17. 2019Spoken Language Identification using ConvNetscited by 6
  18. 2021Comparing Supervised Models And Learned Speech Representations For Classifying Intelligibility Of Disordered Speech On Selected Phrasescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 13; 2024: 5; 2025: 2; 2026: 1.

32 papers.

Sort:

  1. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  2. 2023Transformers in Speech Processing: A Surveycited by 38
  3. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
  4. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 31
  5. 2022TRILLsson: Distilled Universal Paralinguistic Speech Representationscited by 16
  6. 2024HeAR -- Health Acoustic Representationscited by 15
  7. 2023Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation modelcited by 10
  8. 2022BYOL-S: Learning Self-supervised Speech Representations by Bootstrappingcited by 9
  9. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  10. 2023Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settingscited by 8
  11. 2023StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Modelscited by 7
  12. 2022Hybrid Handcrafted and Learnable Audio Representation for Analysis of Speech Under Cognitive and Physical Loadcited by 6
20 more
  1. 2023Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAPcited by 6
  2. 2022Ultra-Low-Bitrate Speech Coding with Pretrained Transformerscited by 5
  3. 2023LanSER: Language-Model Supported Speech Emotion Recognitioncited by 5
  4. 2022Automatic Data Augmentation Selection and Parametrization in Contrastive Self-Supervised Speech Representation Learningcited by 4
  5. 2022The Need for Medically Aware Video Compression in Gastroenterologycited by 3
  6. 2022Learning the joint distribution of two sequences using little or no paired datacited by 3
  7. 2023Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representationscited by 3
  8. 2023Improved Cross-Lingual Transfer Learning For Automatic Speech Translationcited by 2
  9. 2023Introspective Action Advising for Interpretable Transfer Learningcited by 2
  10. 2023Optimizing Audio Augmentations for Contrastive Learning of Health-Related Acoustic Signalscited by 2
  11. 2022Multitask vocal burst modeling with ResNets and pre-trained paralinguistic Conformerscited by 1
  12. 2022Knowledge distillation for fast and accurate DNA sequence correctioncited by 1
  13. 2024Are Paralinguistic Representations all that is needed for Speech Emotion Recognition?cited by 1
  14. 2025Voice Quality Dimensions as Interpretable Primitives for Speaking Style for Atypical Speech and Affectcited by 1
  15. 2022Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devicescited by 0
  16. 2023The unreasonable effectiveness of AI CADe polyp detectors to generalize to new countriescited by 0
  17. 2024Predicting Generalization of AI Colonoscopy Models to Unseen Datacited by 0
  18. 2024Towards Cross-Lingual Audio Abuse Detection in Low-Resource Settings with Few-Shot Learningcited by 0
  19. 2025Meta-Learning Approaches for Speaker-Dependent Voice Fatigue Modelscited by 0
  20. 2026Face and Voice Cross-modal Association with Learning Convex Feature Embeddingcited by 0