mlsanityData to 14 Oct 2026

Automated Audio Captioning with Recurrent Neural Networks

arXiv 1706.10006 ↗First posted 30 Jun 2017cs.SD cs.CL cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

We present the first approach to automated audio captioning. We employ an encoder-decoder scheme with an alignment model in between. The input to the encoder is a sequence of log mel-band energies calculated from an audio file, while the output is a sequence of words, i.e. a caption. The encoder is a multi-layered, bi-directional gated recurrent unit (GRU) and the decoder a multi-layered GRU with a classification layer connected to the last GRU of the decoder. The classification layer and the alignment model are fully connected layers with shared weights between timesteps. The proposed method is evaluated using data drawn from a commercial sound effects library, ProSound Effects. The resulting captions were rated through metrics utilized in machine translation and image captioning fields. Results from metrics show that the proposed method can predict words appearing in the original caption, but not always correctly ordered.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 14; 2023: 5; 2024: 4; 2025: 7; 2026: 1.

40 papers.

Sort:

  1. 2021Audio Captioning Transformercited by 55
  2. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  3. 2023Separate Anything You Describecited by 43
  4. 2021Can Audio Captions Be Evaluated with Image Caption Metrics?cited by 33
  5. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  6. 2021MusCaps: Generating Captions for Music Audiocited by 23
  7. 2022Automated Audio Captioning: An Overview of Recent Progress and New Challengescited by 23
  8. 2022Leveraging Pre-trained BERT for Audio Captioningcited by 18
  9. 2021CL4AC: A Contrastive Loss for Audio Captioningcited by 16
  10. 2022Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learningcited by 15
  11. 2022Visually-Aware Audio Captioning With Adaptive Audio-Visual Attentioncited by 13
  12. 2021Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularizationcited by 12
28 more
  1. 2023Zero-shot audio captioning with audio-language model guidance and audio context keywordscited by 11
  2. 2025TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretrainingcited by 9
  3. 2023Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extractioncited by 6
  4. 2021Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioningcited by 5
  5. 2022Local Information Assisted Attention-free Decoder for Audio Captioningcited by 5
  6. 2022Joint Speech Recognition and Audio Captioningcited by 5
  7. 2024CLAIR-A: Leveraging Large Language Models to Judge Audio Captionscited by 5
  8. 2021Continual Learning for Automated Audio Captioning Using The Learning Without Forgetting Approachcited by 3
  9. 2022Automated Audio Captioning via Fusion of Low- and High- Dimensional Featurescited by 3
  10. 2022Towards Generating Diverse Audio Captions via Adversarial Trainingcited by 3
  11. 2024EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performancecited by 3
  12. 2025CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizercited by 3
  13. 2021Self-training with noisy student model and semi-supervised loss function for dcase 2021 challenge task 4cited by 2
  14. 2022Caption Feature Space Regularization for Audio Captioningcited by 2
  15. 2022Automated Audio Captioning and Language-Based Audio Retrievalcited by 2
  16. 2025Multiple Choice Learning of Low-Rank Adapters for Language Modelingcited by 2
  17. 2021Audio Captioning with Composition of Acoustic and Semantic Informationcited by 1
  18. 2022Text-to-Audio Grounding Based Novel Metric for Evaluating Audio Caption Similaritycited by 1
  19. 2024Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioningcited by 1
  20. 2025Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioningcited by 1
  21. 2025Auditory Intelligence: Understanding the World Through Soundcited by 1
  22. 2025MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Modelscited by 1
  23. 2022Automatic Audio Captioning using Attention weighted Event based Embeddingscited by 0
  24. 2022An investigation on selecting audio pre-trained models for audio captioningcited by 0
  25. 2023Improving Audio Caption Fluency with Automatic Error Correctioncited by 0
  26. 2023Audio Difference Learning for Audio Captioningcited by 0
  27. 2025Better audio representations are more brain-like: linking model-brain alignment with performance in downstream auditory taskscited by 0
  28. 2026An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbationscited by 0