mlsanityData to 14 Oct 2026

VoxServe: Streaming-Centric Serving System for Speech Language Models

arXiv 2602.00269 ↗First posted 30 Jan 2026cs.LG cs.AI cs.DCcited by 3

On the map: Efficient LLM inference and serving → Audio and omni-modal token compression

Abstract (arXiv)

Deploying modern Speech Language Models (SpeechLMs) in streaming settings requires systems that provide low latency, high throughput, and strong guarantees of streamability. Existing systems fall short of supporting diverse models flexibly and efficiently. We present VoxServe, a unified serving system for SpeechLMs that optimizes streaming performance. VoxServe introduces a model-execution abstraction that decouples model architecture from system-level optimizations, thereby enabling support for diverse SpeechLM architectures within a single framework. Building on this abstraction, VoxServe implements streaming-aware scheduling and an asynchronous inference pipeline to improve end-to-end efficiency. Evaluations across multiple modern SpeechLMs show that VoxServe achieves 10-20x higher throughput than existing implementations at comparable latency while maintaining high streaming viability. The code of VoxServe is available at https://github.com/vox-serve/vox-serve.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 44 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

26 of 44 references.

References · 26

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  7. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  8. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  9. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  10. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 224
  11. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 200
  12. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
14 more
  1. 2025Step-Audio 2 Technical Reportcited by 105
  2. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  3. 2024A Survey on Speech Large Language Models for Understandingcited by 71
  4. 2026PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Modelscited by 63
  5. 2021Hi-Fi Multi-Speaker English TTS Datasetcited by 60
  6. 2025Discrete Audio Tokens: More Than a Survey!cited by 38
  7. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 27
  8. 2025MoonCast: High-Quality Zero-Shot Podcast Generationcited by 18
  9. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  10. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  11. 2025LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximationcited by 7
  12. 2025Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generationcited by 6
  13. 2025Toward Low-Latency End-to-End Voice Agents for Telecommunications Using Streaming ASR, Quantized LLMs, and Real-Time TTScited by 5
  14. 2025Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Servingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?cited by 4
  2. 2026M*: A Modular, Extensible, Serving System for Multimodal Modelscited by 3
  3. 2026MURMUR: An Efficient Inference System for Long-Form ASRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Pushing the Limits of On-Device Streaming ASR: A Compact, High-Accuracy English Model for Low-Latency Inference
  2. 2026Beyond Fresh Starts: Stateful Inference for Streaming ASR in Conversational Voice Agents
  3. 2026VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matching
  4. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkit
  5. 2026Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASR
  6. 2026Harnessing Streaming Video in the Wild
  7. 2025Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage
  8. 2026WavSLM: Single-Stream Speech Language Modeling via WavLM Distillation