mlsanityData to 14 Oct 2026

NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis

arXiv 2211.09407 ↗First posted 17 Nov 2022cs.SD eess.AScited by 38

Not on the map. Why?

Abstract (arXiv)

Various applications of voice synthesis have been developed independently despite the fact that they generate "voice" as output in common. In addition, most of the voice synthesis models still require a large number of audio data paired with annotated labels (e.g., text transcription and music score) for training. To this end, we propose a unified framework of synthesizing and manipulating voice signals from analysis features, dubbed NANSY++. The backbone network of NANSY++ is trained in a self-supervised manner that does not require any annotations paired with audio. After training the backbone network, we efficiently tackle four voice applications - i.e. voice conversion, text-to-speech, singing voice synthesis, and voice designing - by partially modeling the analysis features required for each task. Extensive experiments show that the proposed framework offers competitive advantages such as controllability, data efficiency, and fast training convergence, while providing high quality synthesis. Audio samples: tinyurl.com/8tnsy3uc.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Nov 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 19; 2025: 7; 2026: 3.

38 papers.

Sort:

By area:

  1. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  2. 2024Foundation Models for Music: A Surveycited by 37
  3. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  4. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  5. 2024Coding Speech through Vocal Tract Kinematicscited by 18
  6. 2024YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detectioncited by 16
  7. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  8. 2023HiFTNet: A Fast High-Quality Neural Vocoder with Harmonic-plus-Noise Filter and Inverse Short Time Fourier Transformcited by 11
  9. 2023HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfercited by 9
  10. 2024Stutter-Solver: End-to-end Multi-lingual Dysfluency Detectioncited by 8
  11. 2025GenVC: Self-Supervised Zero-Shot Voice Conversioncited by 8
  12. 2025DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesiscited by 8
26 more
  1. 2024VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editingcited by 7
  2. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusioncited by 7
  3. 2024vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoderscited by 6
  4. 2023SelfVC: Voice Conversion With Iterative Refinement using Self Transformationscited by 5
  5. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  6. 2024Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulationcited by 4
  7. 2024Hear Your Face: Face-based voice conversion with F0 estimationcited by 4
  8. 2024Description-based Controllable Text-to-Speech with Cross-Lingual Voice Controlcited by 4
  9. 2025Voice Cloning: Comprehensive Surveycited by 4
  10. 2023Reimagining Speech: A Scoping Review of Deep Learning-Powered Voice Conversioncited by 3
  11. 2023AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignmentcited by 2
  12. 2023A Comparative Study of Voice Conversion Models with Large-Scale Speech and Singing Data: The T13 Systems for the Singing Voice Conversion Challenge 2023cited by 2
  13. 2023ELF: Encoding Speaker-Specific Latent Speech Feature for Speech Synthesiscited by 2
  14. 2024Disentangling Textual and Acoustic Features of Neural Speech Representationscited by 2
  15. 2025SupertonicTTS: Towards Highly Efficient and Streamlined Text-to-Speech Systemcited by 2
  16. 2025Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokenscited by 2
  17. 2024Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Predictioncited by 1
  18. 2025AdaptVC: High Quality Voice Conversion with Adaptive Learningcited by 1
  19. 2025MaskVCT: Masked Voice Codec Transformer for Zero-Shot Voice Conversion With Increased Controllability via Multiple Guidancescited by 1
  20. 2024JenGAN: Stacked Shifted Filters in GAN-Based Speech Synthesiscited by 0
  21. 2024Song Data Cleansing for End-to-End Neural Singer Diarization Using Neural Analysis and Synthesis Frameworkcited by 0
  22. 2024Super Monotonic Alignment Searchcited by 0
  23. 2024CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generationcited by 0
  24. 2026StyleStream: Real-Time Zero-Shot Voice Style Conversioncited by 0
  25. 2026Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbationscited by 0
  26. 2026PASQA: Pitch-Accent-Focused Speech Quality Assessment Model Trained on Synthetic Speech with Accent Errorscited by 0