# Towards Cross-speaker Reading Style Transfer on Audiobook Dataset

arXiv: https://arxiv.org/abs/2208.05359 · HTML: https://mlsanity.com/paper/2208.05359/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 19 references that are in the map.
- First posted on arXiv 10 Aug 2022.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [Robust and fine-grained prosody control of end-to-end speech synthesis](https://mlsanity.com/paper/1811.02122/) (2018)
- [MFA-Conformer: Multi-scale Feature Aggregation Conformer for Automatic Speaker Verification](https://mlsanity.com/paper/2203.15249/) (2022)
- [Fine-grained robust prosody transfer for single-speaker neural text-to-speech](https://mlsanity.com/paper/1907.02479/) (2019)
- [Cross-speaker emotion disentangling and transfer for end-to-end speech synthesis](https://mlsanity.com/paper/2109.06733/) (2021)
- [Towards Multi-Scale Style Control for Expressive Speech Synthesis](https://mlsanity.com/paper/2104.03521/) (2021)
- [Multi-Reference Neural TTS Stylization with Adversarial Cycle Consistency](https://mlsanity.com/paper/1910.11958/) (2019)
- [CAMP: a Two-Stage Approach to Modelling Prosody in Context](https://mlsanity.com/paper/2011.01175/) (2020)
- [Multi-reference Tacotron by Intercross Training for Style Disentangling,Transfer and Control in Speech Synthesis](https://mlsanity.com/paper/1904.02373/) (2019)
- [Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech](https://mlsanity.com/paper/2110.04153/) (2021)
- [The Multi-speaker Multi-style Voice Cloning Challenge 2021](https://mlsanity.com/paper/2104.01818/) (2021)
- [Meta-Voice: Fast few-shot style transfer for expressive voice cloning using meta learning](https://mlsanity.com/paper/2111.07218/) (2021)
- [Multi-speaker Multi-style Text-to-speech Synthesis With Single-speaker Single-style Training Data Scenarios](https://mlsanity.com/paper/2112.12743/) (2021)

## Abstract (arXiv)

Cross-speaker style transfer aims to extract the speech style of the given reference speech, which can be reproduced in the timbre of arbitrary target speakers. Existing methods on this topic have explored utilizing utterance-level style labels to perform style transfer via either global or local scale style representations. However, audiobook datasets are typically characterized by both the local prosody and global genre, and are rarely accompanied by utterance-level style labels. Thus, properly transferring the reading style across different speakers remains a challenging task. This paper aims to introduce a chunk-wise multi-scale cross-speaker style model to capture both the global genre and the local prosody in audiobook speeches. Moreover, by disentangling speaker timbre and style with the proposed switchable adversarial classifiers, the extracted reading style is made adaptable to the timbre of different speakers. Experiment results confirm that the model manages to transfer a given reading style to new target speakers. With the support of local prosody and global genre type predictor, the potentiality of the proposed method in multi-speaker audiobook generation is further revealed.