mlsanityData to 7 Oct 2026

Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities

arXiv 2402.01831 ↗First posted 2 Feb 2024cs.SD cs.LG eess.AScited by 183

Not on the map. Why?

Cited by 31 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Augmenting large language models (LLMs) to understand audio -- including non-speech sounds and non-verbal speech -- is critically important for diverse real-world applications of LLMs. In this paper, we propose Audio Flamingo, a novel audio language model with 1) strong audio understanding abilities, 2) the ability to quickly adapt to unseen tasks via in-context learning and retrieval, and 3) strong multi-turn dialogue abilities. We introduce a series of training techniques, architecture design, and data strategies to enhance our model with these abilities. Extensive evaluations across various audio understanding tasks confirm the efficacy of our method, setting new state-of-the-art benchmarks. Our demo website is https://audioflamingo.github.io/ and the code is open-sourced at https://github.com/NVIDIA/audio-flamingo.

Key facts. Cited by 183 papers in the map (data through 7 Oct 2026). Builds on 69 of its 93 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

69 of 93 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2017Billion-scale similarity search with GPUscited by 2,477
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  6. 2022High Fidelity Neural Audio Compressioncited by 673
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  8. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  9. 2021AST: Audio Spectrogram Transformercited by 634
  10. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  12. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  13. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  14. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  15. 2023MusicLM: Generating Music From Textcited by 442
  16. 2019Clotho: An Audio Captioning Datasetcited by 354
  17. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  18. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  19. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  20. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  21. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  22. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  23. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  24. 2023Listen, Think, and Understandcited by 206
  25. 2016FMA: A Dataset For Music Analysiscited by 205
  26. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  27. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  28. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  29. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  30. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  31. 2023Joint Audio and Speech Understandingcited by 99
  32. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  33. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  34. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  35. 2021The Benefit Of Temporally-Strong Labels In Audio Event Classificationcited by 83
  36. 2023LP-MusicCaps: LLM-Based Pseudo Music Captioningcited by 83
  37. 2023Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioningcited by 77
  38. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 47; 2025: 70; 2026: 66.

183 papers.

Sort:

By area:

  1. 2024Qwen2-Audio Technical Reportcited by 543
  2. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  3. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  4. 2025Kimi-Audio Technical Reportcited by 210
  5. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  6. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  7. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  8. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  9. 2025MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mixcited by 113
  10. 2025Step-Audio 2 Technical Reportcited by 101
  11. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 100
  12. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
38 more
  1. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  2. 2025Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Modelcited by 60
  3. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  4. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  5. 2025Scaling Laws for Native Multimodal Modelscited by 47
  6. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  7. 2024Baichuan-Omni Technical Reportcited by 42
  8. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  9. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignmentcited by 32
  10. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
  11. 2024Jailbreak Attacks and Defenses against Multimodal Generative Models: A Surveycited by 31
  12. 2025SARI: Structured Audio Reasoning via Curriculum-Guided Reinforcement Learningcited by 31
  13. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challengescited by 30
  14. 2024Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Modelscited by 30
  15. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  16. 2024Taming Data and Transformers for Audio Generationcited by 28
  17. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 28
  18. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  19. 2024Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Surveycited by 27
  20. 2024Improving Text-To-Audio Models with Synthetic Captionscited by 19
  21. 2025Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learningcited by 19
  22. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  23. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  24. 2024Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speechcited by 18
  25. 2024Chain-of-Thought Prompting for Speech Translationcited by 17
  26. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  27. 2025CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Followingcited by 15
  28. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 15
  29. 2025MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocolscited by 14
  30. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  31. 2025Scaling Analysis of Interleaved Speech-Text Language Modelscited by 13
  32. 2024ReCLAP: Improving Zero Shot Audio Classification by Describing Soundscited by 12
  33. 2025Guiding Audio Editing with Audio Language Modelcited by 11
  34. 2025UALM: Unified Audio Language Model for Understanding, Generation and Reasoningcited by 10
  35. 2025OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarioscited by 9
  36. 2025OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Modelscited by 9
  37. 2026MOSS-Audio Technical Reportcited by 9
  38. 2024PALM: Few-Shot Prompt Learning for Audio Language Modelscited by 8

50 most cited of 183 All 183 (JSON)