mlsanityData to 19 Oct 2026

Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker Traits and Speech Properties

arXiv 2505.14648 ↗First posted 20 May 2025cs.SD eess.AScited by 1

On the map: Speech and audio language models → Speaker recognition and diarization

Abstract (arXiv)

We introduce Vox-Profile, a comprehensive benchmark to characterize rich speaker traits and speech properties using speech foundation models. Unlike prior work that targets a single dimension of speaker traits, Vox-Profile provides holistic and multi-dimensional profiles spanning both static speaker traits (e.g., age, sex, accent, voice quality) and dynamic speech traits (e.g., emotion, speech flow). This benchmark is grounded in speech science and linguistics, developed with domain experts to accurately index speaker and speech characteristics. We report benchmark experiments using over 15 publicly available speech datasets and several widely used speech foundation models that target various static and dynamic speaker and speech properties. In addition to benchmark experiments, we showcase several downstream applications supported by Vox-Profile. First, we show that Vox-Profile can augment existing speech recognition datasets to analyze ASR performance variability. Vox-Profile is also used as a tool to evaluate the performance of speech generation systems. Finally, we assess the quality of our automated profiles through comparison with human evaluation and show convergent validity. Vox-Profile is publicly available at: https://github.com/tiantiaf0627/vox-profile-release.

Key facts. Cited by 1 paper in the map (data through 19 Oct 2026). Placed in Speaker recognition and diarization, one of 11 directions in Speech and audio language models. Builds on 42 of its 68 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

42 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021VoxCeleb Enrichment for Age and Gender Recognition28 Sep
  2. 2021A Review of Speaker Diarization: Recent Advances with Deep Learning24 Jan

Other references · 40

  1. 2017Attention Is All You Needcited by 61,180
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 2,135
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,596
  7. 2017VoxCeleb: a large-scale speaker identification datasetcited by 985
  8. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 980
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 793
  10. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 680
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 535
  12. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 263
28 more
  1. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 213
  2. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 184
  3. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 183
  4. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 170
  5. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 169
  6. 2023End-to-End Speech Recognition: A Surveycited by 163
  7. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 143
  8. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 117
  9. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 107
  10. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 105
  11. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 101
  12. 2024EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberationcited by 98
  13. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 66
  14. 2021SEP-28k: A Dataset for Stuttering Event Detection From Podcasts With People Who Stuttercited by 59
  15. 2024The VoicePrivacy 2024 Challenge Evaluation Plancited by 48
  16. 2023The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASRcited by 44
  17. 2024GLOBE: A High-quality English Corpus with Global Accents for Zero-shot Speaker Adaptive Text-to-Speechcited by 37
  18. 2024Towards measuring fairness in speech recognition: Fair-Speech datasetcited by 33
  19. 2025Scaling Rich Style-Prompted Text-to-Speech Datasetscited by 28
  20. 2023CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voicecited by 26
  21. 2023PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Modelscited by 21
  22. 2023Speech-based Age and Gender Prediction with Transformerscited by 20
  23. 2024AccentBox: Towards High-Fidelity Zero-Shot Accent Generationcited by 17
  24. 2022Towards Speaker Age Estimation with Label Distribution Learningcited by 15
  25. 2024USAT: A Universal Speaker-Adaptive Text-to-Speech Approachcited by 12
  26. 2024Documenting Ethical Considerations in Open Source AI Modelscited by 5
  27. 2025Demographic Attributes Prediction from Speech Using WavLM Embeddingscited by 5
  28. 2025Tessellated Linear Model for Age Prediction from Voicecited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generationscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2024VoxSim: A perceptual voice similarity dataset
  2. 2024VoxBlink2: A 100K+ Speaker Recognition Corpus and the Open-Set Speaker-Identification Benchmark
  3. 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
  4. 2026Probing Neural Audio Codecs with Speaker Traits and Speech Characteristics
  5. 2025VoxAging: Continuously Tracking Speaker Aging with a Large-Scale Longitudinal Dataset in English and Mandarin
  6. 2021ASR4REAL: An extended benchmark for speech models
  7. 2025Voxlect: A Speech Foundation Model Benchmark for Modeling Dialects and Regional Languages Around the Globe
  8. 2023VoxBlink: A Large Scale Speaker Verification Dataset on Camera