mlsanityData to 7 Oct 2026

Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning

arXiv 2409.01160 ↗First posted 2 Sep 2024eess.AS cs.AI cs.SDcited by 1

Not on the map. Why?

Abstract (arXiv)

In this technical report, we describe our submission to DCASE2024 Challenge Task6 (Automated Audio Captioning) and Task8 (Language-based Audio Retrieval). We develop our approach building upon the EnCLAP audio captioning framework and optimizing it for Task6 of the challenge. Notably, we outline the changes in the underlying components and the incorporation of the reranking process. Additionally, we submit a supplementary retriever model, a byproduct of our modified framework, to Task8. Our proposed systems achieve FENSE score of 0.542 on Task6 and mAP@10 score of 0.386 on Task8, significantly outperforming the baseline models.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

27 of 32 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  7. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  8. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  9. 2022High Fidelity Neural Audio Compressioncited by 673
  10. 2023Simple and Controllable Music Generationcited by 480
  11. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  12. 2019Clotho: An Audio Captioning Datasetcited by 354
15 more
  1. 2022AudioGen: Textually Guided Audio Generationcited by 292
  2. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  3. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  4. 2021Audio Captioning Transformercited by 53
  5. 2017Automated Audio Captioning with Recurrent Neural Networkscited by 40
  6. 2021Can Audio Captions Be Evaluated with Image Caption Metrics?cited by 33
  7. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  8. 2023Prefix tuning for automated audio captioningcited by 26
  9. 2024Codec-SUPERB: An In-Depth Analysis of Sound Codec Modelscited by 25
  10. 2020A Transformer-based Audio Captioning Model with Keyword Estimationcited by 24
  11. 2023Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentationcited by 19
  12. 2023Adapting a ConvNeXt model to audio classification on AudioSetcited by 13
  13. 2023CoNeTTE: An efficient Audio Captioning system leveraging multiple datasets with Task Embeddingcited by 13
  14. 2024Revisiting Deep Audio-Text Retrieval Through the Lens of Transportationcited by 10
  15. 2023Multitask learning in Audio Captioning: a sentence embedding regression loss acts as a regularizercited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answeringcited by 64