# Multi-interaction TTS toward professional recording reproduction

arXiv: https://arxiv.org/abs/2507.00808 · HTML: https://mlsanity.com/paper/2507.00808/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 50 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Builds on (most cited first, 30 of 33)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling](https://mlsanity.com/paper/2401.16380/) (2024)
- [FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation](https://mlsanity.com/paper/2311.01813/) (2023)
- [VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild](https://mlsanity.com/paper/2403.16973/) (2024)
- [PromptTTS: Controllable Text-to-Speech with Text Descriptions](https://mlsanity.com/paper/2211.12171/) (2022)
- [Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow](https://mlsanity.com/paper/1810.00821/) (2018)
- [Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation](https://mlsanity.com/paper/2304.01816/) (2023)
- [Natural language guidance of high-fidelity text-to-speech with synthetic annotations](https://mlsanity.com/paper/2402.01912/) (2024)
- [A Review of Deep Learning Techniques for Speech Processing](https://mlsanity.com/paper/2305.00359/) (2023)
- [Robust and fine-grained prosody control of end-to-end speech synthesis](https://mlsanity.com/paper/1811.02122/) (2018)
- [Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech](https://mlsanity.com/paper/2103.04088/) (2021)
- [Fine-grained robust prosody transfer for single-speaker neural text-to-speech](https://mlsanity.com/paper/1907.02479/) (2019)
- [Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?](https://mlsanity.com/paper/2204.12765/) (2022)
- [Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems](https://mlsanity.com/paper/1512.08756/) (2015)
- [Learning Alignment for Multimodal Emotion Recognition from Speech](https://mlsanity.com/paper/1909.05645/) (2019)
- [Daft-Exprt: Cross-Speaker Prosody Transfer on Any Text for Expressive Speech Synthesis](https://mlsanity.com/paper/2108.02271/) (2021)

## Built on by (most cited first, 2 of 2)

- [Investigation for Relative Voice Impression Estimation](https://mlsanity.com/paper/2602.14172/) (2026)
- [Scalable Direction-Following TTS via Voice Impression-Guided Pseudo Triplet Construction](https://mlsanity.com/paper/2609.02623/) (2026)

## Abstract (arXiv)

Voice directors often iteratively refine voice actors' performances by providing feedback to achieve the desired outcome. While this iterative feedback-based refinement process is important in actual recordings, it has been overlooked in text-to-speech synthesis (TTS). As a result, fine-grained style refinement after the initial synthesis is not possible, even though the synthesized speech often deviates from the user's intended style. To address this issue, we propose a TTS method with multi-step interaction that allows users to intuitively and rapidly refine synthesized speech. Our approach models the interaction between the TTS model and its user to emulate the relationship between voice actors and voice directors. Experiments show that the proposed model with its corresponding dataset enables iterative style refinements in accordance with users' directions, thus demonstrating its multi-interaction capability. Sample audios are available: https://ntt-hilab-gensp.github.io/ssw13multiinteractiontts/