mlsanityData to 7 Oct 2026

Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems

arXiv 2407.13153 ↗First posted 18 Jul 2024cs.CL cs.CR cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

In recent years, there has been increased demand for speech-to-speech translation (S2ST) systems in industry settings. Although successfully commercialized, cloning-based S2ST systems expose their distributors to liabilities when misused by individuals and can infringe on personality rights when exploited by media organizations. This work proposes a regulated S2ST framework called Preset-Voice Matching (PVM). PVM removes cross-lingual voice cloning in S2ST by first matching the input voice to a similar prior consenting speaker voice in the target-language. With this separation, PVM avoids cloning the input speaker, ensuring PVM systems comply with regulations and reduce risk of misuse. Our results demonstrate PVM can significantly improve S2ST system run-time in multi-speaker settings and the naturalness of S2ST synthesized speech. To our knowledge, PVM is the first explicitly regulated S2ST framework leveraging similarly-matched preset-voices for dynamic S2ST tasks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 55 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

21 of 55 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  4. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  5. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  6. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 189
  7. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  8. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 148
  9. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  10. 2018Neural Voice Cloning with a Few Samplescited by 100
  11. 2023Detecting Voice Cloning Attacks via Timbre Watermarkingcited by 39
  12. 2023OpenVoice: Versatile Instant Voice Cloningcited by 30
9 more
  1. 2023Deepfakes, Misinformation, and Disinformation in the Era of Frontier AI, Generative AI, and Large AI Modelscited by 20
  2. 2022Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbingcited by 17
  3. 2022ConferencingSpeech 2022 Challenge: Non-intrusive Objective Speech Quality Assessment (NISQA) Challenge for Online Conferencing Applicationscited by 15
  4. 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 10
  5. 2023Deepfake in the Metaverse: Security Implications for Virtual Gaming, Meetings, and Officescited by 8
  6. 2023Discussion Paper: The Threat of Real Time Deepfakescited by 4
  7. 2022Deep Unfolding for Iterative Stripe Noise Removalcited by 3
  8. 2023A Comparative Study of Pre-trained Speech and Audio Embeddings for Speech Emotion Recognitioncited by 3
  9. 2024GANsemble for Small and Imbalanced Data Sets: A Baseline for Synthetic Microplastics Datacited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Voice "Cloning" is Style Transfercited by 0