# A Transformer-based Audio Captioning Model with Keyword Estimation

arXiv: https://arxiv.org/abs/2007.00222 · HTML: https://mlsanity.com/paper/2007.00222/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2020.

## Built on by (most cited first, 24 of 24)

- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [UnIVAL: Unified Model for Image, Video, Audio and Language Tasks](https://mlsanity.com/paper/2307.16184/) (2023)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning](https://mlsanity.com/paper/2401.17690/) (2024)
- [MusCaps: Generating Captions for Music Audio](https://mlsanity.com/paper/2104.11984/) (2021)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [Leveraging Pre-trained BERT for Audio Captioning](https://mlsanity.com/paper/2203.02838/) (2022)
- [CL4AC: A Contrastive Loss for Audio Captioning](https://mlsanity.com/paper/2107.09990/) (2021)
- [Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention](https://mlsanity.com/paper/2210.16428/) (2022)
- [FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild](https://mlsanity.com/paper/2401.04210/) (2024)
- [AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning](https://mlsanity.com/paper/2407.07801/) (2024)
- [Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning](https://mlsanity.com/paper/2110.07410/) (2021)
- [Local Information Assisted Attention-free Decoder for Audio Captioning](https://mlsanity.com/paper/2201.03217/) (2022)
- [Continual Learning for Automated Audio Captioning Using The Learning Without Forgetting Approach](https://mlsanity.com/paper/2107.08028/) (2021)
- [Automated Audio Captioning via Fusion of Low- and High- Dimensional Features](https://mlsanity.com/paper/2210.05037/) (2022)
- [Towards Generating Diverse Audio Captions via Adversarial Training](https://mlsanity.com/paper/2212.02033/) (2022)
- [EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance](https://mlsanity.com/paper/2409.01201/) (2024)
- [Caption Feature Space Regularization for Audio Captioning](https://mlsanity.com/paper/2204.08409/) (2022)
- [Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning](https://mlsanity.com/paper/2409.01160/) (2024)
- [MAGIC-Enhanced Keyword Prompting for Zero-Shot Audio Captioning with CLIP Models](https://mlsanity.com/paper/2509.12591/) (2025)
- [Automatic Audio Captioning using Attention weighted Event based Embeddings](https://mlsanity.com/paper/2201.12352/) (2022)
- [Automated Audio Captioning with Epochal Difficult Captions for Curriculum Learning](https://mlsanity.com/paper/2206.01918/) (2022)
- [Improving Audio Caption Fluency with Automatic Error Correction](https://mlsanity.com/paper/2306.10090/) (2023)

## Abstract (arXiv)

One of the problems with automated audio captioning (AAC) is the indeterminacy in word selection corresponding to the audio event/scene. Since one acoustic event/scene can be described with several words, it results in a combinatorial explosion of possible captions and difficulty in training. To solve this problem, we propose a Transformer-based audio-captioning model with keyword estimation called TRACKE. It simultaneously solves the word-selection indeterminacy problem with the main task of AAC while executing the sub-task of acoustic event detection/acoustic scene classification (i.e., keyword estimation). TRACKE estimates keywords, which comprise a word set corresponding to audio events/scenes in the input audio, and generates the caption while referring to the estimated keywords to reduce word-selection indeterminacy. Experimental results on a public AAC dataset indicate that TRACKE achieved state-of-the-art performance and successfully estimated both the caption and its keywords.