mlsanityData to 7 Oct 2026

TASTE-Streaming: Towards Streamable Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling

arXiv 2603.12350 ↗First posted 12 Mar 2026cs.CL cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

Text-speech joint spoken language modeling (SLM) aims at natural and intelligent speech-based interactions, but developing such a system may suffer from modality mismatch: speech unit sequences are much longer than text tokens. Prior work reduces this gap with text-aligned tokenization and embedding (TASTE), producing speech tokens that align in lengths with their textual counterparts. However, the dependence on an external ASR system and the use of a non-causal decoder limits streaming use. To address this limitation, we propose TASTE-S, a streamable extension of TASTE suitable for real-time usage. TASTE-S integrates a CTC-based ASR module into the encoder for instant dual-modality encoding. We also redesign the unit decoder to enable on-the-fly decoding. With joint training, we show that TASTE-S matches TASTE's performance while significantly reducing latency. Further investigations reveal that TASTE-S remains robust to transcriptions and enables long-form encoding and decoding.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 33 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

30 of 33 references.

References · 30

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  7. 2022High Fidelity Neural Audio Compressioncited by 673
  8. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  9. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  11. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  12. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
18 more
  1. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  2. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  3. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
  4. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 204
  5. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
  6. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  7. 2023Matcha-TTS: A fast TTS architecture with conditional flow matchingcited by 142
  8. 2022Generative Spoken Dialogue Language Modelingcited by 125
  9. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  10. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  11. 2023Textually Pretrained Speech Language Modelscited by 99
  12. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 95
  13. 2024BigCodec: Pushing the Limits of Low-Bitrate Neural Speech Codeccited by 45
  14. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  15. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 12
  16. 2025Discrete Diffusion for Generative Modeling of Text-Aligned Speech Tokenscited by 7
  17. 2024DM-Codec: Distilling Multimodal Representations for Speech Tokenizationcited by 5
  18. 2025TASLA: Text-Aligned Speech Tokens with Multiple Layer-Aggregationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026StreamAlign: Streaming Text-Aligned Speech Tokenizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment
  2. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
  3. 2025TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining
  4. 2026CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment
  5. 2025DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding
  6. 2026TRADE: Transducer-Augmented Decoder for Speech LLM
  7. 2026FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation
  8. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformers