mlsanityData to 7 Oct 2026

Automated Audio Captioning: An Overview of Recent Progress and New Challenges

arXiv 2205.05949 ↗First posted 12 May 2022eess.AS cs.AI cs.MMcited by 22

Not on the map. Why?

Abstract (arXiv)

Automated audio captioning is a cross-modal translation task that aims to generate natural language descriptions for given audio clips. This task has received increasing attention with the release of freely available datasets in recent years. The problem has been addressed predominantly with deep learning techniques. Numerous approaches have been proposed, such as investigating different neural network architectures, exploiting auxiliary information such as keywords or sentence information to guide caption generation, and employing different training strategies, which have greatly facilitated the development of this field. In this paper, we present a comprehensive review of the published contributions in automated audio captioning, from a variety of existing approaches to evaluation metrics and datasets. We also discuss open challenges and envisage possible future research directions.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 70 of its 109 references that are in the map. First posted on arXiv 12 May 2022.

Builds on

70 of 109 references.

References · 70

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
38 more
  1. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  6. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  7. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  8. 2016Self-critical Sequence Training for Image Captioningcited by 544
  9. 2015An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognitioncited by 519
  10. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  11. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  12. 2019Clotho: An Audio Captioning Datasetcited by 354
  13. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  14. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  15. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  16. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 112
  17. 2018A multi-device dataset for urban acoustic scene classificationcited by 75
  18. 2020A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformercited by 75
  19. 2021Audio Retrieval with Natural Language Queries: A Benchmark Studycited by 61
  20. 2017Towards Diverse and Natural Image Descriptions via a Conditional GANcited by 59
  21. 2021Audio Captioning Transformercited by 53
  22. 2021Sound Event Detection: A Tutorialcited by 51
  23. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  24. 2017Automated Audio Captioning with Recurrent Neural Networkscited by 40
  25. 2019Temporal Reasoning via Audio Question Answeringcited by 38
  26. 2021Can Audio Captions Be Evaluated with Image Caption Metrics?cited by 33
  27. 2022On Metric Learning for Audio-Text Cross-Modal Retrievalcited by 30
  28. 2019Sound Event Detection of Weakly Labelled Data with CNN-Transformer and Automatic Threshold Optimizationcited by 29
  29. 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27
  30. 2014Acoustic Scene Classificationcited by 26
  31. 2021Investigating Local and Global Information for Automated Audio Captioning with Transfer Learningcited by 25
  32. 2020A Transformer-based Audio Captioning Model with Keyword Estimationcited by 24
  33. 2021Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learningcited by 24
  34. 2019Masked Non-Autoregressive Image Captioningcited by 22
  35. 2021An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learningcited by 22
  36. 2019Audio Caption: Listen and Tellcited by 20
  37. 2019Weakly Labelled AudioSet Tagging with Attention Neural Networkscited by 17
  38. 2022Leveraging Pre-trained BERT for Audio Captioningcited by 17

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 4; 2025: 7; 2026: 3.

22 papers.

Sort:

  1. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  2. 2023Separate Anything You Describecited by 43
  3. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 32
  4. 2022Visually-Aware Audio Captioning With Adaptive Audio-Visual Attentioncited by 11
  5. 2025TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretrainingcited by 9
  6. 2023StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Modelscited by 7
  7. 2023Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extractioncited by 6
  8. 2024Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Modelscited by 5
  9. 2025AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioningcited by 5
  10. 2022Towards Generating Diverse Audio Captions via Adversarial Trainingcited by 3
  11. 2025Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioningcited by 3
  12. 2025CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizercited by 3
10 more
  1. 2025Multiple Choice Learning of Low-Rank Adapters for Language Modelingcited by 2
  2. 2024Describe Where You Are: Improving Noise-Robustness for Speech Emotion Recognition with Text Description of the Environmentcited by 1
  3. 2024EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generationcited by 1
  4. 2025AC/DC: LLM-based Audio Comprehension via Dialogue Continuationcited by 1
  5. 2025MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Modelscited by 1
  6. 2023Audio Difference Learning for Audio Captioningcited by 0
  7. 2024AudioSetMix: Enhancing Audio-Language Datasets with LLM-Assisted Augmentationscited by 0
  8. 2026An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbationscited by 0
  9. 2026SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrievalcited by 0
  10. 2026AVSD-Scenes: A Dataset for Audio-Visual Description of Urban Scenescited by 0