# Preset-Voice Matching for Privacy Regulated Speech-to-Speech Translation Systems

arXiv: https://arxiv.org/abs/2407.13153 · HTML: https://mlsanity.com/paper/2407.13153/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 55 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling](https://mlsanity.com/paper/2303.03926/) (2023)
- [StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models](https://mlsanity.com/paper/2306.07691/) (2023)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [Detecting Voice Cloning Attacks via Timbre Watermarking](https://mlsanity.com/paper/2312.03410/) (2023)
- [OpenVoice: Versatile Instant Voice Cloning](https://mlsanity.com/paper/2312.01479/) (2023)
- [Deepfakes, Misinformation, and Disinformation in the Era of Frontier AI, Generative AI, and Large AI Models](https://mlsanity.com/paper/2311.17394/) (2023)
- [Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing](https://mlsanity.com/paper/2212.12137/) (2022)
- [ConferencingSpeech 2022 Challenge: Non-intrusive Objective Speech Quality Assessment (NISQA) Challenge for Online Conferencing Applications](https://mlsanity.com/paper/2203.16032/) (2022)
- [A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation](https://mlsanity.com/paper/2301.10606/) (2023)
- [Deepfake in the Metaverse: Security Implications for Virtual Gaming, Meetings, and Offices](https://mlsanity.com/paper/2303.14612/) (2023)
- [Discussion Paper: The Threat of Real Time Deepfakes](https://mlsanity.com/paper/2306.02487/) (2023)
- [Deep Unfolding for Iterative Stripe Noise Removal](https://mlsanity.com/paper/2209.14973/) (2022)
- [A Comparative Study of Pre-trained Speech and Audio Embeddings for Speech Emotion Recognition](https://mlsanity.com/paper/2304.11472/) (2023)
- [GANsemble for Small and Imbalanced Data Sets: A Baseline for Synthetic Microplastics Data](https://mlsanity.com/paper/2404.07356/) (2024)

## Built on by (most cited first, 1 of 1)

- [Voice "Cloning" is Style Transfer](https://mlsanity.com/paper/2605.16578/) (2026)

## Abstract (arXiv)

In recent years, there has been increased demand for speech-to-speech translation (S2ST) systems in industry settings. Although successfully commercialized, cloning-based S2ST systems expose their distributors to liabilities when misused by individuals and can infringe on personality rights when exploited by media organizations. This work proposes a regulated S2ST framework called Preset-Voice Matching (PVM). PVM removes cross-lingual voice cloning in S2ST by first matching the input voice to a similar prior consenting speaker voice in the target-language. With this separation, PVM avoids cloning the input speaker, ensuring PVM systems comply with regulations and reduce risk of misuse. Our results demonstrate PVM can significantly improve S2ST system run-time in multi-speaker settings and the naturalness of S2ST synthesized speech. To our knowledge, PVM is the first explicitly regulated S2ST framework leveraging similarly-matched preset-voices for dynamic S2ST tasks.