mlsanityData to 19 Oct 2026

FlexiSLM: A Spoken Language Model with Dynamic and Controllable Frame Rates

arXiv 2606.31247 ↗First posted 30 Jun 2026cs.SD eess.AScited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Spoken language models (SLMs) extend LLMs to speech input and output, but existing systems use fixed frame rates (e.g., 25 or 12.5 Hz), overlooking speech's time-varying information density and limiting inference-time quality-speed tradeoffs. Recent dynamic-frame-rate audio tokenizers enable very low average frame rates and controllability, yet had not been applied to SLMs. We introduce FlexiSLM, the first SLM with dynamic, controllable frame rates, using pretrained FlexiCodec for dynamic speech output tokens. It integrates this representation into a multi-task speech-to-speech SLM, extends it with input-side frame compression, and adds direct frame-rate conditioning for accurate control during inference. FlexiSLM outperforms fixed-frame-rate 7B models, including Qwen2.5-Omni and Kimi-Audio, at 12.5 and 6.25 Hz; it can be steered down to 4.0 Hz, and at 6.25 Hz roughly halves inference time relative to 12.5 Hz while retaining strong speech-to-speech quality. Audio samples: https://flexislm.github.io; code and data: https://github.com/AmphionTeam/FlexiSLM.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 47 of its 62 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

47 of 62 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025FlexiCodec: A Dynamic Neural Audio Codec for Low Frame Rates1 Oct
  2. 2025Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding4 Sep
  3. 2025CodecSlime: Temporal Redundancy Compression of Neural Speech Codec via Dynamic Frame Rate26 Jun
  4. 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations11 Jun
  5. 2025Unlocking Temporal Flexibility: Neural Speech Codec with Variable Frame Rate22 May
  6. 2025DualCodec: A Low-Frame-Rate, Semantically-Enhanced Neural Audio Codec for Speech Generation19 May
  7. 2024Investigating Neural Audio Codecs for Speech Language Model-Based Speech Generation6 Sep
  8. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling29 Aug
  9. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens7 Jul
  10. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS26 Jun
  11. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale23 Jun
  12. 2021SoundStream: An End-to-End Neural Audio Codec7 Jul

Other references · 35

  1. 2017Attention Is All You Needcited by 61,180
  2. 2022Training language models to follow instructions with human feedbackcited by 16,145
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,833
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,210
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,661
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,248
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 3,011
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,596
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 872
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 736
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 733
23 more
  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 711
  2. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 680
  3. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 660
  4. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 589
  5. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 498
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 491
  7. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 369
  8. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 365
  9. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 277
  10. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 268
  11. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 245
  12. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 227
  13. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 218
  14. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 171
  15. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 156
  16. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 144
  17. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 104
  18. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 97
  19. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 84
  20. 2023LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Modelscited by 62
  21. 2021SD-QA: Spoken Dialectal Question Answering for the Real Worldcited by 41
  22. 2026Closing the Modality Reasoning Gap for Speech Large Language Modelscited by 12
  23. 2025LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Modelcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Spoken Language Models with Syllabic Speech Tokenization
  2. 2024Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference
  3. 2024Long-Form Speech Generation with Spoken Language Models
  4. 2025SLM-TTA: A Framework for Test-Time Adaptation of Generative Spoken Language Models
  5. 2026Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation
  6. 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
  7. 2026CoSTA: Compressed Speech Token Adaptation for Efficient Spoken Dialogue Models
  8. 2025FlexSpeech: Towards Stable, Controllable and Expressive Text-to-Speech