mlsanityData to 7 Oct 2026

AudioPaLM: A Large Language Model That Can Speak and Listen

arXiv 2306.12925 ↗First posted 22 Jun 2023cs.CL cs.AI cs.SDcited by 333

Not on the map. Why?

Cited by 32 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt. We release examples of our method at https://google-research.github.io/seanet/audiopalm/examples

Key facts. Cited by 333 papers in the map (data through 7 Oct 2026). Builds on 62 of its 78 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

62 of 78 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
38 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  9. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  11. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  13. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  14. 2022High Fidelity Neural Audio Compressioncited by 673
  15. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  16. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  17. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  18. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  19. 2023MusicLM: Generating Music From Textcited by 442
  20. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  21. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  22. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  23. 2022AudioGen: Textually Guided Audio Generationcited by 292
  24. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  25. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
  26. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  27. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  28. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  29. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 167
  30. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 166
  31. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  32. 2021Direct speech-to-speech translation with discrete unitscited by 148
  33. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 125
  34. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  35. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  36. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  37. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  38. 2023Textually Pretrained Speech Language Modelscited by 99

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 42; 2024: 134; 2025: 98; 2026: 59.

333 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 898
  2. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  3. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  4. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  5. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  6. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  7. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  8. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  9. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  10. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
  11. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  12. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
38 more
  1. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  2. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  3. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  4. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 119
  5. 2024Large Language Models for Time Series: A Surveycited by 113
  6. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 107
  7. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  8. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  9. 2025Step-Audio 2 Technical Reportcited by 101
  10. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  11. 2024ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoningcited by 95
  12. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  13. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  14. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  15. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  16. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  17. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  18. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  19. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  20. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  21. 2023Large Language Models for Generative Recommendation: A Survey and Visionary Discussionscited by 72
  22. 2024Best-of-N Jailbreakingcited by 71
  23. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  24. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  25. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  26. 2023AudioChatLlama: Towards General-Purpose Speech Abilities for LLMscited by 59
  27. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  28. 2023LLaSM: Large Language and Speech Modelcited by 56
  29. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  30. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  31. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codeccited by 44
  32. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  33. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 42
  34. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  35. 2023JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Modelscited by 41
  36. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  37. 2024BLSP-Emo: Towards Empathetic Large Speech-Language Modelscited by 36
  38. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35

50 most cited of 333 All 333 (JSON)