mlsanityData to 14 Oct 2026

Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM

arXiv 2409.17353 ↗First posted 25 Sep 2024cs.CLcited by 2

On the map: LLM reasoning → Audio-language model reasoning

Abstract (arXiv)

Current speech-based LLMs are predominantly trained on extensive ASR and TTS datasets, excelling in tasks related to these domains. However, their ability to handle direct speech-to-speech conversations remains notably constrained. These models often rely on an ASR-to-TTS chain-of-thought pipeline, converting speech into text for processing before generating audio responses, which introduces latency and loses audio features. We propose a method that implicitly internalizes ASR chain of thought into a speech LLM, enhancing its native speech understanding capabilities. Our approach reduces latency and improves the model's native understanding of speech, paving the way for more efficient and natural real-time audio interactions. We also release a large-scale synthetic conversational dataset to facilitate further research.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 29 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

26 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming29 Aug

Other references · 25

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  10. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  11. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  12. 2024Qwen2-Audio Technical Reportcited by 560
13 more
  1. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  2. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  4. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 235
  5. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  6. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  7. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 176
  8. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 168
  9. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
  10. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 90
  11. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  12. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoningcited by 60
  13. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generationcited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thoughtcited by 5
  2. 2026Entity Binding Failures in Speech LLM Reasoning: Diagnosis and Chain-of-Thought Interventioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Can Speech LLMs Think while Listening?
  2. 2026Speech LLMs are Contextual Reasoning Transcribers
  3. 2025Chain-of-Thought Training for Open E2E Spoken Dialogue Systems
  4. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  5. 2025Spoken Conversational Agents with Large Language Models
  6. 2026AuRA: Internalizing Audio Understanding into LLMs as LoRA
  7. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation
  8. 2026SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Models