mlsanityData to 7 Oct 2026

Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

arXiv 2305.15255 ↗First posted 24 May 2023cs.CL cs.LG cs.SDcited by 92

Not on the map. Why?

Abstract (arXiv)

We present Spectron, a novel approach to adapting pre-trained large language models (LLMs) to perform spoken question answering (QA) and speech continuation. By endowing the LLM with a pre-trained speech encoder, our model becomes able to take speech inputs and generate speech outputs. The entire system is trained end-to-end and operates directly on spectrograms, simplifying our architecture. Key to our approach is a training objective that jointly supervises speech recognition, text continuation, and speech synthesis using only paired speech-text pairs, enabling a `cross-modal' chain-of-thought within a single decoding pass. Our method surpasses existing spoken language models in speaker preservation and semantic coherence. Furthermore, the proposed model improves upon direct initialization in retaining the knowledge of the original LLM as demonstrated through spoken QA datasets. We release our audio samples (https://michelleramanovich.github.io/spectron/spectron) and spoken QA dataset (https://github.com/google-research-datasets/LLAMA1-Test-Set).

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 49 of its 64 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

49 of 64 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
37 more
  1. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  7. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  8. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  9. 2022High Fidelity Neural Audio Compressioncited by 673
  10. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  12. 2020Jukebox: A Generative Model for Musiccited by 535
  13. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  14. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  15. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  16. 2023MusicLM: Generating Music From Textcited by 442
  17. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  18. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  19. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  20. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  21. 2023Listen, Think, and Understandcited by 206
  22. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  23. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  24. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  25. 2022Generative Spoken Dialogue Language Modelingcited by 125
  26. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  27. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  28. 2023Textually Pretrained Speech Language Modelscited by 99
  29. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  30. 2020Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modelingcited by 65
  31. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  32. 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
  33. 2019Almost Unsupervised Text to Speech and Automatic Speech Recognitioncited by 41
  34. 2022Are discrete units necessary for Spoken Language Modeling?cited by 24
  35. 2022WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iterationcited by 20
  36. 2021Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Modelcited by 14
  37. 2022Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 50; 2026: 20.

92 papers.

Sort:

By area:

  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  2. 2024Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLMcited by 135
  3. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  4. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  5. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 94
  6. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  7. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  8. 2025MinMo: A Multimodal Large Language Model for Seamless Voice Interactioncited by 79
  9. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  10. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 69
  11. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  12. 2024SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Wordscited by 67
38 more
  1. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 44
  2. 2025Voxtralcited by 41
  3. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 33
  4. 2025VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Modelcited by 30
  5. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  6. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 25
  7. 2025STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Modelscited by 25
  8. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
  9. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 22
  10. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  11. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  12. 2025Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Modelcited by 17
  13. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 16
  14. 2025VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generationcited by 16
  15. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  16. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 15
  17. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 15
  18. 2024PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systemscited by 13
  19. 2025OmniEval: A Benchmark for Evaluating Omni-modal Models with Visual, Auditory, and Textual Inputscited by 13
  20. 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 13
  21. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  22. 2025Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Visioncited by 12
  23. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 11
  24. 2025Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Modelscited by 11
  25. 2025JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Modelscited by 11
  26. 2025Aligning Spoken Dialogue Models from User Interactionscited by 11
  27. 2026MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Modelscited by 11
  28. 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
  29. 2024LAST: Language Model Aware Speech Tokenizationcited by 9
  30. 2025FLM-Audio: Natural Monologues Improves Native Full-Duplex Chatbots via Dual Trainingcited by 9
  31. 2025Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Playcited by 8
  32. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 8
  33. 2025MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactionscited by 7
  34. 2025SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgementcited by 7
  35. 2024SpeechCaps: Advancing Instruction-Based Universal Speech Models with Multi-Talker Speaking Style Captioningcited by 6
  36. 2025DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representationscited by 6
  37. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generationcited by 5
  38. 2025RoboEgo System Card: An Omnimodal Model with Native Full Duplexitycited by 5

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs
  2. 2025AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning
  3. 2024Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue
  4. 2023LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models
  5. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Models
  6. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation
  7. 2025End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answering
  8. 2024Speech Retrieval-Augmented Generation without Automatic Speech Recognition