# Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features

arXiv: https://arxiv.org/abs/2212.05805 · HTML: https://mlsanity.com/paper/2212.05805/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 18 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Builds on (most cited first, 16 of 16)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [Translatotron 2: High-quality direct speech-to-speech translation with voice preservation](https://mlsanity.com/paper/2107.08661/) (2021)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Parallel Tacotron: Non-Autoregressive and Controllable TTS](https://mlsanity.com/paper/2010.11439/) (2020)
- [Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesis](https://mlsanity.com/paper/1910.10288/) (2019)
- [Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation](https://mlsanity.com/paper/2204.02967/) (2022)
- [Sequence-to-Sequence Acoustic Modeling for Voice Conversion](https://mlsanity.com/paper/1810.06865/) (2018)
- [Leveraging unsupervised and weakly-supervised data to improve direct speech-to-speech translation](https://mlsanity.com/paper/2203.13339/) (2022)
- [Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation](https://mlsanity.com/paper/2205.08993/) (2022)
- [A Novel Chinese Dialect TTS Frontend with Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2206.04922/) (2022)

## Built on by (most cited first, 1 of 1)

- [Direct Speech-to-Speech Neural Machine Translation: A Survey](https://mlsanity.com/paper/2411.14453/) (2024)

## Abstract (arXiv)

Speech-to-speech translation directly translates a speech utterance to another between different languages, and has great potential in tasks such as simultaneous interpretation. State-of-art models usually contains an auxiliary module for phoneme sequences prediction, and this requires textual annotation of the training dataset. We propose a direct speech-to-speech translation model which can be trained without any textual annotation or content information. Instead of introducing an auxiliary phoneme prediction task in the model, we propose to use bottleneck features as intermediate training objectives for our model to ensure the translation performance of the system. Experiments on Mandarin-Cantonese speech translation demonstrate the feasibility of the proposed approach and the performance can match a cascaded system with respect of translation and synthesis qualities.