# The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge

arXiv: https://arxiv.org/abs/2410.23815 · HTML: https://mlsanity.com/paper/2410.23815/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 29 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [Seed-TTS: A Family of High-Quality Versatile Speech Generation Models](https://mlsanity.com/paper/2406.02430/) (2024)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model](https://mlsanity.com/paper/2406.04904/) (2024)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization](https://mlsanity.com/paper/2404.09956/) (2024)
- [BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data](https://mlsanity.com/paper/2402.08093/) (2024)
- [Better speech synthesis through scaling](https://mlsanity.com/paper/2305.07243/) (2023)
- [SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model](https://mlsanity.com/paper/2104.05557/) (2021)
- [WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark](https://mlsanity.com/paper/2406.05763/) (2024)
- [DiDiSpeech: A Large Scale Mandarin Speech Corpus](https://mlsanity.com/paper/2010.09275/) (2020)
- [Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation](https://mlsanity.com/paper/2406.07422/) (2024)
- [Vec-Tok Speech: speech vectorization and tokenization for neural speech generation](https://mlsanity.com/paper/2310.07246/) (2023)
- [Content-Dependent Fine-Grained Speaker Embedding for Zero-Shot Speaker Adaptation in Text-to-Speech Synthesis](https://mlsanity.com/paper/2204.00990/) (2022)
- [DSPGAN: a GAN-based universal vocoder for high-fidelity TTS by time-frequency domain supervision from DSP](https://mlsanity.com/paper/2211.01087/) (2022)
- [Unet-TTS: Improving Unseen Speaker and Style Transfer in One-shot Voice Cloning](https://mlsanity.com/paper/2109.11115/) (2021)
- [ICAGC 2024: Inspirational and Convincing Audio Generation Challenge 2024](https://mlsanity.com/paper/2407.12038/) (2024)

## Built on by (most cited first, 2 of 2)

- [StreamFlow: Streaming Flow Matching with Block-wise Guided Attention Mask for Speech Token Decoding](https://mlsanity.com/paper/2506.23986/) (2025)
- [HiStyle: Hierarchical Style Embedding Predictor for Text-Prompt-Guided Controllable Speech Synthesis](https://mlsanity.com/paper/2509.25842/) (2025)

## Abstract (arXiv)

This paper presents the NPU-HWC system submitted to the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge 2024 (ICAGC). Our system consists of two modules: a speech generator for Track 1 and a background audio generator for Track 2. In Track 1, we employ Single-Codec to tokenize the speech into discrete tokens and use a language-model-based approach to achieve zero-shot speaking style cloning. The Single-Codec effectively decouples timbre and speaking style at the token level, reducing the acoustic modeling burden on the autoregressive language model. Additionally, we use DSPGAN to upsample 16 kHz mel-spectrograms to high-fidelity 48 kHz waveforms. In Track 2, we propose a background audio generator based on large language models (LLMs). This system produces scene-appropriate accompaniment descriptions, synthesizes background audio with Tango 2, and integrates it with the speech generated by our Track 1 system. Our submission achieves the second place and the first place in Track 1 and Track 2 respectively.