# Knowledge Distillation for Efficient Audio-Visual Video Captioning

arXiv: https://arxiv.org/abs/2306.09947 · HTML: https://mlsanity.com/paper/2306.09947/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 23 of 23)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Dropout Training as Adaptive Regularization](https://mlsanity.com/paper/1307.1493/) (2013)
- [Distilling the Knowledge of BERT for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2008.03822/) (2020)
- [Efficient Video Classification Using Fewer Frames](https://mlsanity.com/paper/1902.10640/) (2019)
- [NeXtVLAD: An Efficient Neural Network to Aggregate Frame-level Features for Large-scale Video Classification](https://mlsanity.com/paper/1811.05014/) (2018)
- [An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning](https://mlsanity.com/paper/2108.02752/) (2021)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [Leveraging Pre-trained BERT for Audio Captioning](https://mlsanity.com/paper/2203.02838/) (2022)
- [Temporal Knowledge Distillation for On-device Audio Classification](https://mlsanity.com/paper/2110.14131/) (2021)
- [Diverse Audio Captioning via Adversarial Training](https://mlsanity.com/paper/2110.06691/) (2021)
- [Knowledge Distillation for Small-footprint Highway Networks](https://mlsanity.com/paper/1608.00892/) (2016)
- [Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention](https://mlsanity.com/paper/2210.16428/) (2022)
- [A Passive Similarity based CNN Filter Pruning for Efficient Acoustic Scene Classification](https://mlsanity.com/paper/2203.15751/) (2022)
- [Simple Pooling Front-ends For Efficient Audio Classification](https://mlsanity.com/paper/2210.00943/) (2022)
- [Channel Compression: Rethinking Information Redundancy among Channels in CNN Architecture](https://mlsanity.com/paper/2007.01696/) (2020)
- [Label Denoising with Large Ensembles of Heterogeneous Neural Networks](https://mlsanity.com/paper/1809.04403/) (2018)
- [Low-complexity CNNs for Acoustic Scene Classification](https://mlsanity.com/paper/2207.11529/) (2022)
- [Automated Audio Captioning via Fusion of Low- and High- Dimensional Features](https://mlsanity.com/paper/2210.05037/) (2022)

## Abstract (arXiv)

Automatically describing audio-visual content with texts, namely video captioning, has received significant attention due to its potential applications across diverse fields. Deep neural networks are the dominant methods, offering state-of-the-art performance. However, these methods are often undeployable in low-power devices like smartphones due to the large size of the model parameters. In this paper, we propose to exploit simple pooling front-end and down-sampling algorithms with knowledge distillation for audio and visual attributes using a reduced number of audio-visual frames. With the help of knowledge distillation from the teacher model, our proposed method greatly reduces the redundant information in audio-visual streams without losing critical contexts for caption generation. Extensive experimental evaluations on the MSR-VTT dataset demonstrate that our proposed approach significantly reduces the inference time by about 80% with a small sacrifice (less than 0.02%) in captioning accuracy.