mlsanityData to 7 Oct 2026

Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks

arXiv 2309.07937 ↗First posted 14 Sep 2023eess.AS cs.LG cs.SDcited by 71

Not on the map. Why?

Abstract (arXiv)

We propose a decoder-only language model, VoxtLM, that can perform four tasks: speech recognition, speech synthesis, text generation, and speech continuation. VoxtLM integrates text vocabulary with discrete speech tokens from self-supervised speech features and uses special tokens to enable multitask learning. Compared to a single-task model, VoxtLM exhibits a significant improvement in speech synthesis, with improvements in both speech intelligibility from 28.9 to 5.6 and objective quality from 2.68 to 3.90. VoxtLM also improves speech generation and speech recognition performance over the single-task counterpart. Further, VoxtLM is trained with publicly available data and training recipes and model checkpoints are open-sourced to make fully reproducible work.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 35 of its 40 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

35 of 40 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  11. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  12. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
23 more
  1. 2022High Fidelity Neural Audio Compressioncited by 673
  2. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  3. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  4. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  5. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  6. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  7. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  8. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  9. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  10. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  11. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  12. 2023End-to-End Speech Recognition: A Surveycited by 127
  13. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 126
  14. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  15. 2023Textually Pretrained Speech Language Modelscited by 99
  16. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  17. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  18. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  19. 2021Generalization Ability of MOS Prediction Networkscited by 69
  20. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  21. 2023Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learningcited by 28
  22. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 21
  23. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 41; 2025: 21; 2026: 6.

71 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  2. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  3. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  4. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  5. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  6. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 69
  7. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 48
  8. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  9. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  10. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  11. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  12. 2024Salmon: A Suite for Acoustic Language Model Evaluationcited by 24
38 more
  1. 2024Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedbackcited by 22
  2. 2024Preference Alignment Improves Language Model-Based TTScited by 21
  3. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  4. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  5. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  6. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  7. 2024Decoder-only Architecture for Streaming End-to-end Speech Recognitioncited by 12
  8. 2024MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservationcited by 11
  9. 2024dMel: Speech Tokenization made Simplecited by 11
  10. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  11. 2024DiscreteSLU: A Large Language Model with Self-Supervised Discrete Speech Units for Spoken Language Understandingcited by 10
  12. 2024Efficient Streaming LLM for Speech Recognitioncited by 10
  13. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  14. 2025ESPnet-SpeechLM: An Open Speech Language Model Toolkitcited by 9
  15. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  16. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 9
  17. 2024Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representationcited by 8
  18. 2024Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approachcited by 8
  19. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 8
  20. 2025Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Playcited by 8
  21. 2025From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLMcited by 7
  22. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  23. 2024Task Arithmetic for Language Expansion in Speech Translationcited by 6
  24. 2024Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistantcited by 6
  25. 2024VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuningcited by 6
  26. 2024Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processingcited by 5
  27. 2024FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Modelcited by 5
  28. 2024Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generationcited by 5
  29. 2024Mamba-based Decoder-Only Approach with Bidirectional Speech Modeling for Speech Recognitioncited by 5
  30. 2025Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modelingcited by 5
  31. 2024TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languagescited by 4
  32. 2024Multi-Modal Retrieval For Large Language Model Based Speech Recognitioncited by 4
  33. 2024On the Effectiveness of Acoustic BPE in Decoder-Only TTScited by 4
  34. 2025NoLoCo: No-all-reduce Low Communication Training Method for Large Modelscited by 4
  35. 2026Bagpiper: Solving Open-Ended Audio Tasks via Rich Captionscited by 4
  36. 2024SpeechComposer: Unifying Multiple Speech Tasks with Prompt Compositioncited by 3
  37. 2024SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Datacited by 3
  38. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3

50 most cited of 71 All 71 (JSON)