Multi-interaction TTS toward professional recording reproduction
Not on the map. Why?
Abstract (arXiv)
Voice directors often iteratively refine voice actors' performances by providing feedback to achieve the desired outcome. While this iterative feedback-based refinement process is important in actual recordings, it has been overlooked in text-to-speech synthesis (TTS). As a result, fine-grained style refinement after the initial synthesis is not possible, even though the synthesized speech often deviates from the user's intended style. To address this issue, we propose a TTS method with multi-step interaction that allows users to intuitively and rapidly refine synthesized speech. Our approach models the interaction between the TTS model and its user to emulate the relationship between voice actors and voice directors. Experiments show that the proposed model with its corresponding dataset enables iterative style refinements in accordance with users' directions, thus demonstrating its multi-interaction capability. Sample audios are available: https://ntt-hilab-gensp.github.io/ssw13multiinteractiontts/
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 1 Jul 2025.
Builds on
33 of 50 references.
References · 33
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
- 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
- 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
21 more
- 2018Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesiscited by 259
- 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
- 2021A Survey on Neural Speech Synthesiscited by 184
- 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
- 2023FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generationcited by 122
- 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
- 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
- 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 105
- 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 68
- 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
- 2023A Review of Deep Learning Techniques for Speech Processingcited by 52
- 2018Robust and fine-grained prosody control of end-to-end speech synthesiscited by 48
- 2021Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speechcited by 37
- 2019Fine-grained robust prosody transfer for single-speaker neural text-to-speechcited by 27
- 2022Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?cited by 27
- 2015Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problemscited by 23
- 2019Learning Alignment for Multimodal Emotion Recognition from Speechcited by 22
- 2021Daft-Exprt: Cross-Speaker Prosody Transfer on Any Text for Expressive Speech Synthesiscited by 10
- 2023Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation modelcited by 10
- 2024Voice Attribute Editing with Text Promptcited by 9
- 2021Model architectures to extrapolate emotional expressions in DNN-based text-to-speechcited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.