# Voice Impression Control in Zero-Shot TTS

arXiv: https://arxiv.org/abs/2506.05688 · HTML: https://mlsanity.com/paper/2506.05688/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Speech Resynthesis from Discrete Disentangled Self-Supervised Representations](https://mlsanity.com/paper/2104.00355/) (2021)
- [LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models](https://mlsanity.com/paper/2305.13655/) (2023)
- [NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models](https://mlsanity.com/paper/2403.03100/) (2024)
- [PromptTTS: Controllable Text-to-Speech with Text Descriptions](https://mlsanity.com/paper/2211.12171/) (2022)
- [Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow](https://mlsanity.com/paper/1810.00821/) (2018)
- [InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt](https://mlsanity.com/paper/2301.13662/) (2023)
- [Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddings](https://mlsanity.com/paper/1910.10838/) (2019)
- [Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech](https://mlsanity.com/paper/2103.04088/) (2021)
- [Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?](https://mlsanity.com/paper/2204.12765/) (2022)
- [Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems](https://mlsanity.com/paper/1512.08756/) (2015)
- [Cross-speaker emotion disentangling and transfer for end-to-end speech synthesis](https://mlsanity.com/paper/2109.06733/) (2021)
- [EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vector](https://mlsanity.com/paper/2411.02625/) (2024)
- [LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning](https://mlsanity.com/paper/2406.07969/) (2024)
- [ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models](https://mlsanity.com/paper/2305.13831/) (2023)
- [Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model](https://mlsanity.com/paper/2304.11976/) (2023)
- [Cross-speaker Style Transfer with Prosody Bottleneck in Neural Speech Synthesis](https://mlsanity.com/paper/2107.12562/) (2021)
- [Model architectures to extrapolate emotional expressions in DNN-based text-to-speech](https://mlsanity.com/paper/2102.10345/) (2021)
- [Robust speaker recognition using unsupervised adversarial invariance](https://mlsanity.com/paper/1911.00940/) (2019)
- [Speech Synthesis along Perceptual Voice Quality Dimensions](https://mlsanity.com/paper/2501.08791/) (2025)
- [Controllable Speaking Styles Using a Large Language Model](https://mlsanity.com/paper/2305.10321/) (2023)

## Built on by (most cited first, 3 of 3)

- [Investigation for Relative Voice Impression Estimation](https://mlsanity.com/paper/2602.14172/) (2026)
- [Scalable Direction-Following TTS via Voice Impression-Guided Pseudo Triplet Construction](https://mlsanity.com/paper/2609.02623/) (2026)
- [ReaFlow-TTS: Realization-Conditioned Flow Matching for High-Quality and Controllable Speech Synthesis](https://mlsanity.com/paper/2609.28906/) (2026)

## Abstract (arXiv)

Para-/non-linguistic information in speech is pivotal in shaping the listeners' impression. Although zero-shot text-to-speech (TTS) has achieved high speaker fidelity, modulating subtle para-/non-linguistic information to control perceived voice characteristics, i.e., impressions, remains challenging. We have therefore developed a voice impression control method in zero-shot TTS that utilizes a low-dimensional vector to represent the intensities of various voice impression pairs (e.g., dark-bright). The results of both objective and subjective evaluations have demonstrated our method's effectiveness in impression control. Furthermore, generating this vector via a large language model enables target-impression generation from a natural language description of the desired impression, thus eliminating the need for manual optimization. Audio examples are available on our demo page (https://ntt-hilab-gensp.github.io/is2025voiceimpression/).