# Diverse Audio Captioning via Adversarial Training

arXiv: https://arxiv.org/abs/2110.06691 · HTML: https://mlsanity.com/paper/2110.06691/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2021.

## Built on by (most cited first, 14 of 14)

- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [Leveraging Pre-trained BERT for Audio Captioning](https://mlsanity.com/paper/2203.02838/) (2022)
- [On the Audio Hallucinations in Large Audio-Video Language Models](https://mlsanity.com/paper/2401.09774/) (2024)
- [Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention](https://mlsanity.com/paper/2210.16428/) (2022)
- [Local Information Assisted Attention-free Decoder for Audio Captioning](https://mlsanity.com/paper/2201.03217/) (2022)
- [AMA-LSTM: Pioneering Robust and Fair Financial Audio Analysis for Stock Volatility Prediction](https://mlsanity.com/paper/2407.18324/) (2024)
- [Towards Generating Diverse Audio Captions via Adversarial Training](https://mlsanity.com/paper/2212.02033/) (2022)
- [Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglement](https://mlsanity.com/paper/2308.11923/) (2023)
- [Multiple Choice Learning of Low-Rank Adapters for Language Modeling](https://mlsanity.com/paper/2507.10419/) (2025)
- [Text-to-Audio Grounding Based Novel Metric for Evaluating Audio Caption Similarity](https://mlsanity.com/paper/2210.06354/) (2022)
- [Efficient Audio Captioning with Encoder-Level Knowledge Distillation](https://mlsanity.com/paper/2407.14329/) (2024)
- [Extremely Simple Out-of-distribution Detection for Audio-visual Generalized Zero-shot Learning](https://mlsanity.com/paper/2503.22197/) (2025)
- [Knowledge Distillation for Efficient Audio-Visual Video Captioning](https://mlsanity.com/paper/2306.09947/) (2023)

## Abstract (arXiv)

Audio captioning aims at generating natural language descriptions for audio clips automatically. Existing audio captioning models have shown promising improvement in recent years. However, these models are mostly trained via maximum likelihood estimation (MLE),which tends to make captions generic, simple and deterministic. As different people may describe an audio clip from different aspects using distinct words and grammars, we argue that an audio captioning system should have the ability to generate diverse captions for a fixed audio clip and across similar audio clips. To address this problem, we propose an adversarial training framework for audio captioning based on a conditional generative adversarial network (C-GAN), which aims at improving the naturalness and diversity of generated captions. Unlike processing data of continuous values in a classical GAN, a sentence is composed of discrete tokens and the discrete sampling process is non-differentiable. To address this issue, policy gradient, a reinforcement learning technique, is used to back-propagate the reward to the generator. The results show that our proposed model can generate more diverse captions, as compared to state-of-the-art methods.