mlsanityData to 14 Oct 2026

Prefix tuning for automated audio captioning

arXiv 2303.17489 ↗First posted 30 Mar 2023eess.AS cs.MM cs.SDcited by 27

Not on the map. Why?

Abstract (arXiv)

Audio captioning aims to generate text descriptions from environmental sounds. One challenge of audio captioning is the difficulty of the generalization due to the lack of audio-text paired training data. In this work, we propose a simple yet effective method of dealing with small-scaled datasets by leveraging a pre-trained language model. We keep the language model frozen to maintain the expressivity for text generation, and we only learn to extract global and temporal features from the input audio. To bridge a modality gap between the audio features and the language model, we employ mapping networks that translate audio features to the continuous vectors the language model can understand, called prefixes. We evaluate our proposed method on the Clotho and AudioCaps dataset and show our method outperforms prior arts in diverse experimental settings.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Mar 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 16; 2025: 4; 2026: 2.

27 papers.

Sort:

By area:

  1. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  2. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 80
  3. 2023Separate Anything You Describecited by 43
  4. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  5. 2023RECAP: Retrieval-Augmented Audio Captioningcited by 27
  6. 2024Learning Spatially-Aware Language and Audio Embeddingscited by 17
  7. 2024Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reactioncited by 13
  8. 2024DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioningcited by 11
  9. 2024FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wildcited by 9
  10. 2023Weakly-supervised Automated Audio Captioning via text only trainingcited by 8
  11. 2025LAVCap: LLM-based Audio-Visual Captioning using Optimal Transportcited by 7
  12. 2024AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioningcited by 6
15 more
  1. 2023Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Modelscited by 5
  2. 2024A Multimodal Approach to Device-Directed Speech Detection with Large Language Modelscited by 5
  3. 2024Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignmentcited by 5
  4. 2024EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performancecited by 3
  5. 2024Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learningcited by 3
  6. 2025Diffusion-Link: Diffusion Probabilistic Model for Bridging the Audio-Text Modality Gapcited by 3
  7. 2024Towards Diverse and Efficient Audio Captioning via Diffusion Modelscited by 2
  8. 2024Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictionscited by 1
  9. 2024Efficient Audio Captioning with Encoder-Level Knowledge Distillationcited by 1
  10. 2024Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioningcited by 1
  11. 2025Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioningcited by 1
  12. 2026LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergencecited by 1
  13. 2024S2Cap: A Benchmark and a Baseline for Singing Style Captioningcited by 0
  14. 2025Classifier-Guided Captioning Across Modalitiescited by 0
  15. 2026The Language of Touch: Translating Vibrations into Text with Dual-Branch Learningcited by 0