# Automated Audio Captioning with Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1706.10006 · HTML: https://mlsanity.com/paper/1706.10006/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2017.

## Built on by (most cited first, 30 of 40)

- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [Can Audio Captions Be Evaluated with Image Caption Metrics?](https://mlsanity.com/paper/2110.04684/) (2021)
- [EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning](https://mlsanity.com/paper/2401.17690/) (2024)
- [MusCaps: Generating Captions for Music Audio](https://mlsanity.com/paper/2104.11984/) (2021)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [Leveraging Pre-trained BERT for Audio Captioning](https://mlsanity.com/paper/2203.02838/) (2022)
- [CL4AC: A Contrastive Loss for Audio Captioning](https://mlsanity.com/paper/2107.09990/) (2021)
- [Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learning](https://mlsanity.com/paper/2203.15526/) (2022)
- [Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention](https://mlsanity.com/paper/2210.16428/) (2022)
- [Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularization](https://mlsanity.com/paper/2108.04692/) (2021)
- [Zero-shot audio captioning with audio-language model guidance and audio context keywords](https://mlsanity.com/paper/2311.08396/) (2023)
- [TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretraining](https://mlsanity.com/paper/2505.07609/) (2025)
- [Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction](https://mlsanity.com/paper/2310.07284/) (2023)
- [Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning](https://mlsanity.com/paper/2110.07410/) (2021)
- [Local Information Assisted Attention-free Decoder for Audio Captioning](https://mlsanity.com/paper/2201.03217/) (2022)
- [Joint Speech Recognition and Audio Captioning](https://mlsanity.com/paper/2202.01405/) (2022)
- [CLAIR-A: Leveraging Large Language Models to Judge Audio Captions](https://mlsanity.com/paper/2409.12962/) (2024)
- [Continual Learning for Automated Audio Captioning Using The Learning Without Forgetting Approach](https://mlsanity.com/paper/2107.08028/) (2021)
- [Automated Audio Captioning via Fusion of Low- and High- Dimensional Features](https://mlsanity.com/paper/2210.05037/) (2022)
- [Towards Generating Diverse Audio Captions via Adversarial Training](https://mlsanity.com/paper/2212.02033/) (2022)
- [EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance](https://mlsanity.com/paper/2409.01201/) (2024)
- [CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer](https://mlsanity.com/paper/2506.00800/) (2025)
- [Self-training with noisy student model and semi-supervised loss function for dcase 2021 challenge task 4](https://mlsanity.com/paper/2107.02569/) (2021)
- [Caption Feature Space Regularization for Audio Captioning](https://mlsanity.com/paper/2204.08409/) (2022)
- [Automated Audio Captioning and Language-Based Audio Retrieval](https://mlsanity.com/paper/2207.04156/) (2022)
- [Multiple Choice Learning of Low-Rank Adapters for Language Modeling](https://mlsanity.com/paper/2507.10419/) (2025)
- [Audio Captioning with Composition of Acoustic and Semantic Information](https://mlsanity.com/paper/2105.06355/) (2021)
- [Text-to-Audio Grounding Based Novel Metric for Evaluating Audio Caption Similarity](https://mlsanity.com/paper/2210.06354/) (2022)

## Abstract (arXiv)

We present the first approach to automated audio captioning. We employ an encoder-decoder scheme with an alignment model in between. The input to the encoder is a sequence of log mel-band energies calculated from an audio file, while the output is a sequence of words, i.e. a caption. The encoder is a multi-layered, bi-directional gated recurrent unit (GRU) and the decoder a multi-layered GRU with a classification layer connected to the last GRU of the decoder. The classification layer and the alignment model are fully connected layers with shared weights between timesteps. The proposed method is evaluated using data drawn from a commercial sound effects library, ProSound Effects. The resulting captions were rated through metrics utilized in machine translation and image captioning fields. Results from metrics show that the proposed method can predict words appearing in the original caption, but not always correctly ordered.