mlsanityData to 14 Oct 2026

HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer

arXiv 2307.16171 ↗First posted 30 Jul 2023cs.SD cs.AI cs.MMcited by 9

Not on the map. Why?

Abstract (arXiv)

Despite rapid progress in the voice style transfer (VST) field, recent zero-shot VST systems still lack the ability to transfer the voice style of a novel speaker. In this paper, we present HierVST, a hierarchical adaptive end-to-end zero-shot VST model. Without any text transcripts, we only use the speech dataset to train the model by utilizing hierarchical variational inference and self-supervised representation. In addition, we adopt a hierarchical adaptive generator that generates the pitch representation and waveform audio sequentially. Moreover, we utilize unconditional generation to improve the speaker-relative acoustic capacity in the acoustic representation. With a hierarchical adaptive structure, the model can adapt to a novel voice style and convert speech progressively. The experimental results demonstrate that our method outperforms other VST models in zero-shot VST scenarios. Audio samples are available at urlhttps://hiervst.github.io/.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 30 Jul 2023.

Builds on

25 of 34 references.

References · 25

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  3. 2022High Fidelity Neural Audio Compressioncited by 687
  4. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  5. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
  6. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  7. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  8. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 192
  9. 2021Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Schemecited by 100
  10. 2021Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generationcited by 93
  11. 2021Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representationscited by 81
  12. 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 74
13 more
  1. 2022NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesiscited by 38
  2. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversioncited by 38
  3. 2022SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shapingcited by 37
  4. 2020F0-consistent many-to-many non-parallel voice conversion via conditional autoencodercited by 35
  5. 2022Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Datacited by 35
  6. 2020Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesiscited by 28
  7. 2021Fre-GAN: Adversarial Frequency-consistent Audio Synthesiscited by 24
  8. 2022ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speechcited by 22
  9. 2020The ins and outs of speaker recognition: lessons from VoxSRC 2020cited by 21
  10. 2022DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoderscited by 15
  11. 2022Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversioncited by 14
  12. 2022Source-Filter HiFi-GAN: Fast and Pitch Controllable High-Fidelity Neural Vocodercited by 11
  13. 2021Reinforce-Aligner: Reinforcement Alignment Search for Robust End-to-End Text-to-Speechcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 6.

9 papers.

  1. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  2. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 24
  3. 2025From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speechcited by 9
  4. 2024DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignmentcited by 8
  5. 2025VoicePrompter: Robust Zero-Shot Voice Conversion with Voice Prompt and Conditional Flow Matchingcited by 3
  6. 2025Towards Better Disentanglement in Non-Autoregressive Zero-Shot Expressive Voice Conversioncited by 2
  7. 2025When Humans Growl and Birds Speak: High-Fidelity Voice Conversion from Human to Animal and Designed Soundscited by 1
  8. 2025GSA-TTS : Toward Zero-Shot Speech Synthesis based on Gradual Style Adaptorcited by 0
  9. 2025MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglementcited by 0