The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge
Not on the map. Why?
Abstract (arXiv)
This paper presents the NPU-HWC system submitted to the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge 2024 (ICAGC). Our system consists of two modules: a speech generator for Track 1 and a background audio generator for Track 2. In Track 1, we employ Single-Codec to tokenize the speech into discrete tokens and use a language-model-based approach to achieve zero-shot speaking style cloning. The Single-Codec effectively decouples timbre and speaking style at the token level, reducing the acoustic modeling burden on the autoregressive language model. Additionally, we use DSPGAN to upsample 16 kHz mel-spectrograms to high-fidelity 48 kHz waveforms. In Track 2, we propose a background audio generator based on large language models (LLMs). This system produces scene-appropriate accompaniment descriptions, synthesizes background audio with Tango 2, and integrates it with the speech generated by our Track 1 system. Our submission achieves the second place and the first place in Track 1 and Track 2 respectively.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 29 references that are in the map. First posted on arXiv 31 Oct 2024.
Builds on
26 of 29 references.
References · 26
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
- 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
- 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
- 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
- 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
- 2022AudioGen: Textually Guided Audio Generationcited by 295
- 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
- 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
14 more
- 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 179
- 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
- 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimizationcited by 97
- 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
- 2023Better speech synthesis through scalingcited by 63
- 2021SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Modelcited by 56
- 2024WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmarkcited by 40
- 2020DiDiSpeech: A Large Scale Mandarin Speech Corpuscited by 33
- 2024Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generationcited by 23
- 2023Vec-Tok Speech: speech vectorization and tokenization for neural speech generationcited by 12
- 2022Content-Dependent Fine-Grained Speaker Embedding for Zero-Shot Speaker Adaptation in Text-to-Speech Synthesiscited by 10
- 2022DSPGAN: a GAN-based universal vocoder for high-fidelity TTS by time-frequency domain supervision from DSPcited by 6
- 2021Unet-TTS: Improving Unseen Speaker and Style Transfer in One-shot Voice Cloningcited by 5
- 2024ICAGC 2024: Inspirational and Convincing Audio Generation Challenge 2024cited by 1
Built on by
Citing papers per year: 2025: 2.
2 papers.