mlsanityData to 7 Oct 2026

Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrieval

arXiv 2012.07331 ↗First posted 14 Dec 2020eess.AS cs.CL cs.SDcited by 27

Not on the map. Why?

Abstract (arXiv)

The goal of audio captioning is to translate input audio into its description using natural language. One of the problems in audio captioning is the lack of training data due to the difficulty in collecting audio-caption pairs by crawling the web. In this study, to overcome this problem, we propose to use a pre-trained large-scale language model. Since an audio input cannot be directly inputted into such a language model, we utilize guidance captions retrieved from a training dataset based on similarities that may exist in different audio. Then, the caption of the audio input is generated by using a pre-trained language model while referring to the guidance captions. Experimental results show that (i) the proposed method has succeeded to use a pre-trained language model for audio captioning, and (ii) the oracle performance of the pre-trained model-based caption generator was clearly better than that of the conventional method trained from scratch.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 4; 2023: 9; 2024: 4; 2025: 5; 2026: 1.

27 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  3. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  4. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  5. 2021Audio Captioning Transformercited by 53
  6. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 47
  7. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  8. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  9. 2023RECAP: Retrieval-Augmented Audio Captioningcited by 26
  10. 2021MusCaps: Generating Captions for Music Audiocited by 23
  11. 2022Automated Audio Captioning: An Overview of Recent Progress and New Challengescited by 22
  12. 2025WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Modelscited by 18
15 more
  1. 2025RANGE: Retrieval Augmented Neural Fields for Multi-Resolution Geo-Embeddingscited by 18
  2. 2022Leveraging Pre-trained BERT for Audio Captioningcited by 17
  3. 2021Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularizationcited by 12
  4. 2023Zero-shot audio captioning with audio-language model guidance and audio context keywordscited by 11
  5. 2021Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioningcited by 5
  6. 2025CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizercited by 3
  7. 2023Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglementcited by 2
  8. 2024R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generationcited by 2
  9. 2026Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perceptioncited by 2
  10. 2024Intelligent Director: An Automatic Framework for Dynamic Visual Composition using ChatGPTcited by 1
  11. 2025MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Modelscited by 1
  12. 2022Automatic Audio Captioning using Attention weighted Event based Embeddingscited by 0
  13. 2022Automated Audio Captioning with Epochal Difficult Captions for Curriculum Learningcited by 0
  14. 2023Efficient Audio Captioning Transformer with Patchout and Text Guidancecited by 0
  15. 2023Improving Audio Caption Fluency with Automatic Error Correctioncited by 0