# AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning

arXiv: https://arxiv.org/abs/2407.07801 · HTML: https://mlsanity.com/paper/2407.07801/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [MAViL: Masked Audio-Video Learners](https://mlsanity.com/paper/2212.08071/) (2022)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [Audiovisual Masked Autoencoders](https://mlsanity.com/paper/2212.05922/) (2022)
- [Prefix tuning for automated audio captioning](https://mlsanity.com/paper/2303.17489/) (2023)
- [A Transformer-based Audio Captioning Model with Keyword Estimation](https://mlsanity.com/paper/2007.00222/) (2020)
- [Audio Caption: Listen and Tell](https://mlsanity.com/paper/1902.09254/) (2019)
- [CL4AC: A Contrastive Loss for Audio Captioning](https://mlsanity.com/paper/2107.09990/) (2021)
- [Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learning](https://mlsanity.com/paper/2203.15526/) (2022)
- [Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention](https://mlsanity.com/paper/2210.16428/) (2022)
- [Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularization](https://mlsanity.com/paper/2108.04692/) (2021)
- [Local Information Assisted Attention-free Decoder for Audio Captioning](https://mlsanity.com/paper/2201.03217/) (2022)
- [Automated Audio Captioning via Fusion of Low- and High- Dimensional Features](https://mlsanity.com/paper/2210.05037/) (2022)
- [Dual Transformer Decoder based Features Fusion Network for Automated Audio Captioning](https://mlsanity.com/paper/2305.18753/) (2023)

## Built on by (most cited first, 6 of 6)

- [Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation](https://mlsanity.com/paper/2506.23271/) (2025)
- [LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport](https://mlsanity.com/paper/2501.09291/) (2025)
- [Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning](https://mlsanity.com/paper/2505.22045/) (2025)
- [Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model](https://mlsanity.com/paper/2505.13062/) (2025)
- [LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2601.04658/) (2026)
- [Listening without Looking: Modality Bias in Audio-Visual Captioning](https://mlsanity.com/paper/2510.24024/) (2025)

## Abstract (arXiv)

In recent years, advancements in representation learning and language models have propelled Automated Captioning (AC) to new heights, enabling the generation of human-level descriptions. Leveraging these advancements, we propose AVCap, an Audio-Visual Captioning framework, a simple yet powerful baseline approach applicable to audio-visual captioning. AVCap utilizes audio-visual features as text tokens, which has many advantages not only in performance but also in the extensibility and scalability of the model. AVCap is designed around three pivotal dimensions: the exploration of optimal audio-visual encoder architectures, the adaptation of pre-trained models according to the characteristics of generated text, and the investigation into the efficacy of modality fusion in captioning. Our method outperforms existing audio-visual captioning methods across all metrics and the code is available on https://github.com/JongSuk1/AVCap