# Efficient Audio Captioning with Encoder-Level Knowledge Distillation

arXiv: https://arxiv.org/abs/2407.14329 · HTML: https://mlsanity.com/paper/2407.14329/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 19 Jul 2024.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 19 of 19)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation](https://mlsanity.com/paper/2211.04772/) (2022)
- [Prefix tuning for automated audio captioning](https://mlsanity.com/paper/2303.17489/) (2023)
- [Investigating Local and Global Information for Automated Audio Captioning with Transfer Learning](https://mlsanity.com/paper/2102.11457/) (2021)
- [Diverse Audio Captioning via Adversarial Training](https://mlsanity.com/paper/2110.06691/) (2021)
- [Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention](https://mlsanity.com/paper/2210.16428/) (2022)
- [Beyond the Status Quo: A Contemporary Survey of Advances and Challenges in Audio Captioning](https://mlsanity.com/paper/2205.05357/) (2022)
- [Parameter Efficient Audio Captioning With Faithful Guidance Using Audio-text Shared Latent Representation](https://mlsanity.com/paper/2309.03340/) (2023)
- [Enhance Temporal Relations in Audio Captioning with Sound Event Detection](https://mlsanity.com/paper/2306.01533/) (2023)
- [Multi-Level Knowledge Distillation for Speech Emotion Recognition in Noisy Conditions](https://mlsanity.com/paper/2312.13556/) (2023)
- [Efficient Similarity-based Passive Filter Pruning for Compressing CNNs](https://mlsanity.com/paper/2210.17416/) (2022)

## Built on by (most cited first, 1 of 1)

- [Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoders](https://mlsanity.com/paper/2501.03038/) (2025)

## Abstract (arXiv)

Significant improvement has been achieved in automated audio captioning (AAC) with recent models. However, these models have become increasingly large as their performance is enhanced. In this work, we propose a knowledge distillation (KD) framework for AAC. Our analysis shows that in the encoder-decoder based AAC models, it is more effective to distill knowledge into the encoder as compared with the decoder. To this end, we incorporate encoder-level KD loss into training, in addition to the standard supervised loss and sequence-level KD loss. We investigate two encoder-level KD methods, based on mean squared error (MSE) loss and contrastive loss, respectively. Experimental results demonstrate that contrastive KD is more robust than MSE KD, exhibiting superior performance in data-scarce situations. By leveraging audio-only data into training in the KD framework, our student model achieves competitive performance, with an inference speed that is 19 times fasterfootnoteAn online demo is available at urlhttps://huggingface.co/spaces/wsntxxn/efficientaudiocaptioning.