mlsanityData to 14 Oct 2026

VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation

arXiv 2305.16107 ↗First posted 25 May 2023cs.CL cs.SD eess.AScited by 87

Not on the map. Why?

Abstract (arXiv)

Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities. In this paper, we propose VioLA, a single auto-regressive Transformer decoder-only network that unifies various cross-modal tasks involving speech and text, such as speech-to-text, text-to-text, text-to-speech, and speech-to-speech tasks, as a conditional codec language model task via multi-task learning framework. To accomplish this, we first convert all the speech utterances to discrete tokens (similar to the textual data) using an offline neural codec encoder. In such a way, all these tasks are converted to token-based sequence conversion problems, which can be naturally handled with one conditional language model. We further integrate task IDs (TID) and language IDs (LID) into the proposed model to enhance the modeling capability of handling different languages and tasks. Experimental results demonstrate that the proposed VioLA model can support both single-modal and cross-modal tasks well, and the decoder-only model achieves a comparable and even better performance than the strong baselines.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

28 of 36 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  11. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  12. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
16 more
  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  2. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 717
  3. 2022High Fidelity Neural Audio Compressioncited by 687
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  5. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  6. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  7. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  8. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  9. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  10. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  11. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  12. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  13. 2023End-to-End Speech Recognition: A Surveycited by 129
  14. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 97
  15. 2021Deep Learning Based Assessment of Synthetic Speech Naturalnesscited by 35
  16. 2021SpeechNet: A Universal Modularized Model for Speech Processing Taskscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 58; 2025: 11; 2026: 5.

87 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 361
  3. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  4. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  5. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
  6. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  7. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  8. 2024Large Language Models for Time Series: A Surveycited by 114
  9. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  10. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  11. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  12. 2024Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agentscited by 72
38 more
  1. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 66
  2. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  3. 2024Language Model Can Listen While Speakingcited by 63
  4. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  5. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codeccited by 44
  6. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 41
  7. 2024VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignmentcited by 38
  8. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 38
  9. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 37
  10. 2024BLSP-Emo: Towards Empathetic Large Speech-Language Modelscited by 36
  11. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  12. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  13. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  14. 2023Speech Translation with Large Language Models: An Industrial Practicecited by 28
  15. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  16. 2024RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesiscited by 26
  17. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
  18. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  19. 2023Boosting Large Language Model for Speech Synthesis: An Empirical Studycited by 21
  20. 2025Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Modelscited by 21
  21. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
  22. 2023Towards human-like spoken dialogue generation between AI agents from written dialoguecited by 19
  23. 2024Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generatorscited by 18
  24. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 16
  25. 2024CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversationscited by 15
  26. 2024Comparing Discrete and Continuous Space LLMs for Speech Recognitioncited by 15
  27. 2024Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversationcited by 14
  28. 2024PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systemscited by 14
  29. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 14
  30. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  31. 2024LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Modelscited by 12
  32. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 11
  33. 2024dMel: Speech Tokenization made Simplecited by 11
  34. 2023Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognitioncited by 10
  35. 2024TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservationcited by 10
  36. 2024Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformercited by 10
  37. 2024BLSP-KD: Bootstrapping Language-Speech Pre-training via Knowledge Distillationcited by 9
  38. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9

50 most cited of 87 All 87 (JSON)