mlsanityData to 7 Oct 2026

Efficient Parallel Audio Generation using Group Masked Language Modeling

arXiv 2401.01099 ↗First posted 2 Jan 2024eess.AS cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We present a fast and high-quality codec language model for parallel audio generation. While SoundStorm, a state-of-the-art parallel audio generation model, accelerates inference speed compared to autoregressive models, it still suffers from slow inference due to iterative sampling. To resolve this problem, we propose Group-Masked Language Modeling~(G-MLM) and Group Iterative Parallel Decoding~(G-IPD) for efficient parallel audio generation. Both the training and sampling schemes enable the model to synthesize high-quality audio with a small number of iterations by effectively modeling the group-wise conditional dependencies. In addition, our model employs a cross-attention-based architecture to capture the speaker style of the prompt voice and improves computational efficiency. Experimental results demonstrate that our proposed model outperforms the baselines in prompt-based audio generation.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 33 references that are in the map. First posted on arXiv 2 Jan 2024.

Builds on

32 of 33 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  12. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
20 more
  1. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  2. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  3. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  4. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  5. 2022High Fidelity Neural Audio Compressioncited by 673
  6. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  8. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  9. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  10. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  11. 2023Simple and Controllable Music Generationcited by 480
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  13. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 233
  14. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  15. 2021Unsupervised Speech Recognitioncited by 147
  16. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  17. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codeccited by 95
  18. 2023VampNet: Music Generation via Masked Acoustic Token Modelingcited by 57
  19. 2022Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpuscited by 10
  20. 2023Stack-and-Delay: a new codebook pattern for music generationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2024MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Modelscited by 32
  2. 2025Voice Cloning: Comprehensive Surveycited by 4
  3. 2024SegINR: Segment-wise Implicit Neural Representation for Sequence Alignment in Neural Text-to-Speechcited by 0
  4. 2025MusRec: Zero-Shot Text-to-Music Editing via Rectified Flow and Diffusion Transformerscited by 0
  5. 2026FlowSonic: Stable Zero-Shot Music Editing via High-Order Trajectory Integrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling
  2. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding
  3. 2026An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation
  4. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
  5. 2025Continuous Audio Language Models
  6. 2026UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
  7. 2026Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation ↗
  8. 2024TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization