mlsanityData to 7 Oct 2026

CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASR

arXiv 2411.07607 ↗First posted 12 Nov 2024eess.AS cs.LG cs.SDcited by 2

Not on the map. Why?

Abstract (arXiv)

CTC compressor can be an effective approach to integrate audio encoders to decoder-only models, which has gained growing interest for different speech applications. In this work, we propose a novel CTC compressor based joint speech and text training (CJST) framework for decoder-only ASR. CJST matches speech and text modalities from both directions by exploring a simple modality adaptor and several features of the CTC compressor, including sequence compression, on-the-fly forced peaky alignment and CTC class embeddings. Experimental results on the Librispeech and TED-LIUM2 corpora show that the proposed CJST achieves an effective text injection without the need of duration handling, leading to the best performance for both in-domain and cross-domain scenarios. We also provide a comprehensive study on CTC compressor, covering various compression modes, edge case handling and behavior under both clean and noisy data conditions, which reveals the most robust setting to use CTC compressor for decoder-only models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

25 of 31 references.

References · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2015Listen, Attend and Spellcited by 542
  9. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  10. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  11. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  12. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
13 more
  1. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  2. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  3. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  4. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  5. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  6. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  7. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  8. 2021CTC-based Compression for Direct Speech Translationcited by 45
  9. 2021Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognitioncited by 37
  10. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  11. 2022Integrating Text Inputs For Training and Adapting RNN Transducer ASR Modelscited by 20
  12. 2024Decoder-only Architecture for Streaming End-to-end Speech Recognitioncited by 12
  13. 2023Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognitioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025PAC: Pronunciation-Aware Contextualized Large Language Model-based Automatic Speech Recognitioncited by 1
  2. 2026Text-Utilization for Encoder-dominated Speech Recognition Modelscited by 0