mlsanityData to 14 Oct 2026

Multilingual Multi-Speaker Unit Vocoders: A Systematic Analysis of Discrete Speech Representations

arXiv 2606.06740 ↗First posted 4 Jun 2026cs.SD cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Discrete speech units obtained via k-means clustering of self supervised embeddings entangle phonetic, speaker, and language information, causing speaker mixing and cross-lingual interference in multilingual multi-speaker speech generation. Despite growing use in Audio LLMs and speech to speech systems, unit vocoders remain underexplored. We analyze a BigVGAN based unit vocoder, across four Indian languages. We study the interaction between cluster size and conditioning strategies using WER, speaker similarity, and unit level metrics. Results show that cluster size governs intelligibility by improving phonetic discriminability, while explicit speaker conditioning is indispensable for preventing identity collapse. Language supervision yields further gains mainly at lower cluster sizes where units remain ambiguous. Our analysis shows similar phonemes across languages collapse to the same cluster IDs at smaller inventories, with larger clusters progressively separating them.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 20 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

19 of 20 references.

References · 19

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  3. 2016Least Squares Generative Adversarial Networkscited by 1,001
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  5. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  6. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  7. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  8. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  9. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  10. 2021ECAPA-TDNN Embeddings for Speaker Diarizationcited by 47
  11. 2024IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languagescited by 41
  12. 2024VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Musiccited by 30
7 more
  1. 2022Towards Building Text-To-Speech Systems for the Next Billion Userscited by 27
  2. 2024StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learningcited by 21
  3. 2022Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languagescited by 16
  4. 2024IndicVoices-R: Unlocking a Massive Multilingual Multi-speaker Speech Corpus for Scaling Indian TTScited by 16
  5. 2023SPRING-INX: A Multilingual Indian Language Speech Corpus by SPRING Lab, IIT Madrascited by 11
  6. 2022data2vec-aqc: Search for the right Teaching Assistant in the Teacher-Student training setupcited by 5
  7. 2023Multilingual Speech-to-Speech Translation into Multiple Target Languagescited by 5

Built on by

Not cited yet.