mlsanityData to 7 Oct 2026

EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning

arXiv 2401.17690 ↗First posted 31 Jan 2024eess.AS cs.AI cs.SDcited by 28

Not on the map. Why?

Abstract (arXiv)

We propose EnCLAP, a novel framework for automated audio captioning. EnCLAP employs two acoustic representation models, EnCodec and CLAP, along with a pretrained language model, BART. We also introduce a new training objective called masked codec modeling that improves acoustic awareness of the pretrained language model. Experimental results on AudioCaps and Clotho demonstrate that our model surpasses the performance of baseline models. Source code will be available at https://github.com/jaeyeonkim99/EnCLAP . An online demo is available at https://huggingface.co/spaces/enclap-team/enclap .

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 25 of its 32 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

25 of 32 references.

References · 25

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2022A ConvNet for the 2020scited by 3,545
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  6. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  7. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  8. 2019Clotho: An Audio Captioning Datasetcited by 354
  9. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  10. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  11. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  12. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 112
13 more
  1. 2021Audio Captioning Transformercited by 53
  2. 2017Automated Audio Captioning with Recurrent Neural Networkscited by 40
  3. 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27
  4. 2023Prefix tuning for automated audio captioningcited by 26
  5. 2020A Transformer-based Audio Captioning Model with Keyword Estimationcited by 24
  6. 2022Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learningcited by 15
  7. 2023Adapting a ConvNeXt model to audio classification on AudioSetcited by 13
  8. 2021Improving the Performance of Automated Audio Captioning via Integrating the Acoustic and Semantic Informationcited by 11
  9. 2022Exploring Train and Test-Time Augmentations for Audio-Language Learningcited by 11
  10. 2022Local Information Assisted Attention-free Decoder for Audio Captioningcited by 5
  11. 2023Graph Attention for Automated Audio Captioningcited by 3
  12. 2023Multitask learning in Audio Captioning: a sentence embedding regression loss acts as a regularizercited by 3
  13. 2023Dual Transformer Decoder based Features Fusion Network for Automated Audio Captioningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 11; 2026: 5.

28 papers.

Sort:

By area:

  1. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  2. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  3. 2024Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoningcited by 29
  4. 2024Taming Data and Transformers for Audio Generationcited by 28
  5. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  6. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12
  7. 2024DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioningcited by 10
  8. 2024Scaling Concept With Text-Guided Diffusion Modelscited by 8
  9. 2024Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Datacited by 7
  10. 2025LAVCap: LLM-based Audio-Visual Captioning using Optimal Transportcited by 7
  11. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 7
  12. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
16 more
  1. 2024C3LLM: Conditional Multimodal Content Generation Using Large Language Modelscited by 3
  2. 2024EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performancecited by 3
  3. 2024Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learningcited by 3
  4. 2025CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizercited by 3
  5. 2025Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gapcited by 3
  6. 2024Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioningcited by 1
  7. 2025Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoderscited by 1
  8. 2025Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioningcited by 1
  9. 2025Do Joint Language-Audio Embeddings Encode Perceptual Timbre Semantics?cited by 1
  10. 2026LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergencecited by 1
  11. 2025EGGCodec: A Robust Neural Encodec Framework for EGG Reconstruction and F0 Extractioncited by 0
  12. 2025Audio-Guided Visual Editing with Complex Multi-Modal Promptscited by 0
  13. 2025Semantic-Aware Confidence Calibration for Automated Audio Captioningcited by 0
  14. 2026SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Trainingcited by 0
  15. 2026Robust Audio-Text Retrieval via Cross-Modal Attention and Hybrid Losscited by 0
  16. 2026CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioningcited by 0