mlsanityData to 14 Oct 2026

Pengi: An Audio Language Model for Audio Tasks

arXiv 2305.11834 ↗First posted 19 May 2023eess.AS cs.SDcited by 220

Not on the map. Why?

Cited by 32 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In the domain of audio processing, Transfer Learning has facilitated the rise of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches have led to the development of versatile models capable of tackling a wide array of tasks, while delivering state-of-the-art performance. However, current models inherently lack the capacity to produce the requisite language for open-ended tasks, such as Audio Captioning or Audio Question & Answering. We introduce Pengi, a novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text-generation tasks. It takes as input, an audio recording, and text, and generates free-form text as output. The input audio is represented as a sequence of continuous embeddings by an audio encoder. A text encoder does the same for the corresponding text input. Both sequences are combined as a prefix to prompt a pre-trained frozen language model. The unified architecture of Pengi enables open-ended tasks and close-ended tasks without any additional fine-tuning or task-specific extensions. When evaluated on 22 downstream tasks, our approach yields state-of-the-art performance in several of them. Our results show that connecting language models with audio models is a major step towards general-purpose audio understanding

Key facts. Cited by 220 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 May 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 77; 2025: 63; 2026: 61.

220 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Modelscited by 909
  2. 2025Qwen2.5-Omni Technical Reportcited by 736
  3. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  4. 2024Qwen2-Audio Technical Reportcited by 560
  5. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  6. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  7. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  8. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 186
  9. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  10. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 156
  11. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  12. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 128
38 more
  1. 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
  2. 2025Step-Audio 2 Technical Reportcited by 105
  3. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  4. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  5. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  6. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  7. 2025Baichuan-Audio: A Unified Framework for End-to-End Speech Interactioncited by 68
  8. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  9. 2024Empowering LLMs to Understand and Generate Complex Vector Graphicscited by 66
  10. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  11. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 59
  12. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  13. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  14. 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
  15. 2024MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Modelscited by 44
  16. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  17. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 42
  18. 2024Baichuan-Omni Technical Reportcited by 42
  19. 2025Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detectorcited by 40
  20. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 39
  21. 2024AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Modelscited by 38
  22. 2024Foundation Models for Music: A Surveycited by 37
  23. 2025CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answeringcited by 37
  24. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 36
  25. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  26. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  27. 2024Revisiting Multi-modal Emotion Learning with Broad State Space Models and Probability-guidance Fusioncited by 33
  28. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 33
  29. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 31
  30. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 31
  31. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 31
  32. 2023LLark: A Multimodal Instruction-Following Language Model for Musiccited by 29
  33. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  34. 2024Taming Data and Transformers for Audio Generationcited by 28
  35. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?cited by 25
  36. 2024MaskSR: Masked Language Model for Full-band Speech Restorationcited by 24
  37. 2024Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?cited by 23
  38. 2025A Survey of Interactive Generative Videocited by 23

50 most cited of 220 All 220 (JSON)