# EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance

arXiv: https://arxiv.org/abs/2409.01201 · HTML: https://mlsanity.com/paper/2409.01201/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 23 of 23)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [Automated Audio Captioning with Recurrent Neural Networks](https://mlsanity.com/paper/1706.10006/) (2017)
- [Can Audio Captions Be Evaluated with Image Caption Metrics?](https://mlsanity.com/paper/2110.04684/) (2021)
- [EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning](https://mlsanity.com/paper/2401.17690/) (2024)
- [Prefix tuning for automated audio captioning](https://mlsanity.com/paper/2303.17489/) (2023)
- [Codec-SUPERB: An In-Depth Analysis of Sound Codec Models](https://mlsanity.com/paper/2402.13071/) (2024)
- [A Transformer-based Audio Captioning Model with Keyword Estimation](https://mlsanity.com/paper/2007.00222/) (2020)
- [Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentation](https://mlsanity.com/paper/2309.17352/) (2023)
- [Adapting a ConvNeXt model to audio classification on AudioSet](https://mlsanity.com/paper/2306.00830/) (2023)
- [CoNeTTE: An efficient Audio Captioning system leveraging multiple datasets with Task Embedding](https://mlsanity.com/paper/2309.00454/) (2023)
- [Exploring Train and Test-Time Augmentations for Audio-Language Learning](https://mlsanity.com/paper/2210.17143/) (2022)
- [Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation](https://mlsanity.com/paper/2405.10084/) (2024)
- [Multitask learning in Audio Captioning: a sentence embedding regression loss acts as a regularizer](https://mlsanity.com/paper/2305.01482/) (2023)

## Built on by (most cited first, 3 of 3)

- [Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering](https://mlsanity.com/paper/2503.11197/) (2025)
- [CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer](https://mlsanity.com/paper/2506.00800/) (2025)
- [Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models](https://mlsanity.com/paper/2606.15751/) (2026)

## Abstract (arXiv)

In this work, we aim to analyze and optimize the EnCLAP framework, a state-of-the-art model in automated audio captioning. We investigate the impact of modifying the acoustic encoder components, explore pretraining with different dataset scales, and study the effectiveness of a reranking scheme. Through extensive experimentation and quantitative analysis of generated captions, we develop EnCLAP++, an enhanced version that significantly surpasses the original.