# Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention

arXiv: https://arxiv.org/abs/2210.16428 · HTML: https://mlsanity.com/paper/2210.16428/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [Separate What You Describe: Language-Queried Audio Source Separation](https://mlsanity.com/paper/2203.15147/) (2022)
- [Automated Audio Captioning with Recurrent Neural Networks](https://mlsanity.com/paper/1706.10006/) (2017)
- [On Metric Learning for Audio-Text Cross-Modal Retrieval](https://mlsanity.com/paper/2203.15537/) (2022)
- [Investigating Local and Global Information for Automated Audio Captioning with Transfer Learning](https://mlsanity.com/paper/2102.11457/) (2021)
- [A Transformer-based Audio Captioning Model with Keyword Estimation](https://mlsanity.com/paper/2007.00222/) (2020)
- [Personalized Dialogue Generation with Persona-Adaptive Attention](https://mlsanity.com/paper/2210.15088/) (2022)
- [An Encoder-Decoder Based Audio Captioning System With Transfer and Reinforcement Learning](https://mlsanity.com/paper/2108.02752/) (2021)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [Leveraging Pre-trained BERT for Audio Captioning](https://mlsanity.com/paper/2203.02838/) (2022)
- [CL4AC: A Contrastive Loss for Audio Captioning](https://mlsanity.com/paper/2107.09990/) (2021)
- [Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learning](https://mlsanity.com/paper/2203.15526/) (2022)
- [Diverse Audio Captioning via Adversarial Training](https://mlsanity.com/paper/2110.06691/) (2021)
- [An Attempt towards Interpretable Audio-Visual Video Captioning](https://mlsanity.com/paper/1812.02872/) (2018)
- [Audiovisual transfer learning for audio tagging and sound event detection](https://mlsanity.com/paper/2106.05408/) (2021)
- [Visually-aware Acoustic Event Detection using Heterogeneous Graphs](https://mlsanity.com/paper/2207.07935/) (2022)
- [Automated Audio Captioning via Fusion of Low- and High- Dimensional Features](https://mlsanity.com/paper/2210.05037/) (2022)

## Built on by (most cited first, 13 of 13)

- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Separate Anything You Describe](https://mlsanity.com/paper/2308.05037/) (2023)
- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [On the Audio Hallucinations in Large Audio-Video Language Models](https://mlsanity.com/paper/2401.09774/) (2024)
- [FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild](https://mlsanity.com/paper/2401.04210/) (2024)
- [LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport](https://mlsanity.com/paper/2501.09291/) (2025)
- [AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning](https://mlsanity.com/paper/2407.07801/) (2024)
- [Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning](https://mlsanity.com/paper/2505.22045/) (2025)
- [HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals](https://mlsanity.com/paper/2507.13318/) (2025)
- [Efficient Audio Captioning with Encoder-Level Knowledge Distillation](https://mlsanity.com/paper/2407.14329/) (2024)
- [Knowledge Distillation for Efficient Audio-Visual Video Captioning](https://mlsanity.com/paper/2306.09947/) (2023)
- [HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning](https://mlsanity.com/paper/2508.06475/) (2025)
- [Listening without Looking: Modality Bias in Audio-Visual Captioning](https://mlsanity.com/paper/2510.24024/) (2025)

## Abstract (arXiv)

Audio captioning aims to generate text descriptions of audio clips. In the real world, many objects produce similar sounds. How to accurately recognize ambiguous sounds is a major challenge for audio captioning. In this work, inspired by inherent human multimodal perception, we propose visually-aware audio captioning, which makes use of visual information to help the description of ambiguous sounding objects. Specifically, we introduce an off-the-shelf visual encoder to extract video features and incorporate the visual features into an audio captioning system. Furthermore, to better exploit complementary audio-visual contexts, we propose an audio-visual attention mechanism that adaptively integrates audio and visual context and removes the redundant information in the latent space. Experimental results on AudioCaps, the largest audio captioning dataset, show that our proposed method achieves state-of-the-art results on machine translation metrics.