# CL4AC: A Contrastive Loss for Audio Captioning

arXiv: https://arxiv.org/abs/2107.09990 · HTML: https://mlsanity.com/paper/2107.09990/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [Improved Image Captioning via Policy Gradient optimization of SPIDEr](https://mlsanity.com/paper/1612.00370/) (2016)
- [Automated Audio Captioning with Recurrent Neural Networks](https://mlsanity.com/paper/1706.10006/) (2017)
- [A Transformer-based Audio Captioning Model with Keyword Estimation](https://mlsanity.com/paper/2007.00222/) (2020)
- [Token-Level Supervised Contrastive Learning for Punctuation Restoration](https://mlsanity.com/paper/2107.09099/) (2021)
- [Effects of Word-frequency based Pre- and Post- Processings for Audio Captioning](https://mlsanity.com/paper/2009.11436/) (2020)
- [WaveTransformer: A Novel Architecture for Audio Captioning Based on Learning Temporal and Time-Frequency Information](https://mlsanity.com/paper/2010.11098/) (2020)

## Built on by (most cited first, 16 of 16)

- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [Conformer-Based Self-Supervised Learning for Non-Speech Audio Tasks](https://mlsanity.com/paper/2110.07313/) (2021)
- [Leveraging Pre-trained BERT for Audio Captioning](https://mlsanity.com/paper/2203.02838/) (2022)
- [Fine-grained Audible Video Description](https://mlsanity.com/paper/2303.15616/) (2023)
- [Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learning](https://mlsanity.com/paper/2203.15526/) (2022)
- [Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention](https://mlsanity.com/paper/2210.16428/) (2022)
- [ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification](https://mlsanity.com/paper/2211.13189/) (2022)
- [Zero-shot audio captioning with audio-language model guidance and audio context keywords](https://mlsanity.com/paper/2311.08396/) (2023)
- [AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning](https://mlsanity.com/paper/2407.07801/) (2024)
- [Local Information Assisted Attention-free Decoder for Audio Captioning](https://mlsanity.com/paper/2201.03217/) (2022)
- [Automated Audio Captioning via Fusion of Low- and High- Dimensional Features](https://mlsanity.com/paper/2210.05037/) (2022)
- [Towards Generating Diverse Audio Captions via Adversarial Training](https://mlsanity.com/paper/2212.02033/) (2022)
- [Caption Feature Space Regularization for Audio Captioning](https://mlsanity.com/paper/2204.08409/) (2022)
- [Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning](https://mlsanity.com/paper/2502.05435/) (2025)

## Abstract (arXiv)

Automated Audio captioning (AAC) is a cross-modal translation task that aims to use natural language to describe the content of an audio clip. As shown in the submissions received for Task 6 of the DCASE 2021 Challenges, this problem has received increasing interest in the community. The existing AAC systems are usually based on an encoder-decoder architecture, where the audio signal is encoded into a latent representation, and aligned with its corresponding text descriptions, then a decoder is used to generate the captions. However, training of an AAC system often encounters the problem of data scarcity, which may lead to inaccurate representation and audio-text alignment. To address this problem, we propose a novel encoder-decoder framework called Contrastive Loss for Audio Captioning (CL4AC). In CL4AC, the self-supervision signals derived from the original audio-text paired data are used to exploit the correspondences between audio and texts by contrasting samples, which can improve the quality of latent representation and the alignment between audio and texts, while trained with limited data. Experiments are performed on the Clotho dataset to show the effectiveness of our proposed approach.