mlsanityData to 7 Oct 2026

An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation

arXiv 2607.02119 ↗First posted 2 Jul 2026eess.AS cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 32 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

30 of 32 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  8. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  9. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  12. 2023Simple and Controllable Music Generationcited by 480
18 more
  1. 2025Qwen3-Omni Technical Reportcited by 451
  2. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  3. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  4. 20242 OLMo 2 Furiouscited by 302
  5. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  6. 2025Kimi-Audio Technical Reportcited by 210
  7. 2024Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Modelcited by 70
  8. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  9. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 47
  10. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  11. 2025Discrete Audio Tokens: More Than a Survey!cited by 37
  12. 2024ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speechcited by 25
  13. 2024Is Flash Attention Stable?cited by 10
  14. 2025UALM: Unified Audio Language Model for Understanding, Generation and Reasoningcited by 10
  15. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 9
  16. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  17. 2026Bagpiper: Solving Open-Ended Audio Tasks via Rich Captionscited by 4
  18. 2025BSCodec: A Band-Split Neural Codec for High-Quality Universal Audio Reconstructioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Make Some Noise: Towards LLM audio reasoning and generation using sound tokens
  2. 2026TRADE: Transducer-Augmented Decoder for Speech LLM
  3. 2026FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation
  4. 2026Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs
  5. 2025From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training
  6. 2026SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Models
  7. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers
  8. 2025MGM-Omni: Scaling Omni LLMs to Personalized Long-Horizon Speech