# Listening without Looking: Modality Bias in Audio-Visual Captioning

arXiv: https://arxiv.org/abs/2510.24024 · HTML: https://mlsanity.com/paper/2510.24024/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Builds on (most cited first, 15 of 15)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Improved Image Captioning via Policy Gradient optimization of SPIDEr](https://mlsanity.com/paper/1612.00370/) (2016)
- [Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention](https://mlsanity.com/paper/2210.16428/) (2022)
- [LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport](https://mlsanity.com/paper/2501.09291/) (2025)
- [AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning](https://mlsanity.com/paper/2407.07801/) (2024)
- [Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning](https://mlsanity.com/paper/2505.22045/) (2025)
- [Language-Guided Contrastive Audio-Visual Masked Autoencoder with Automatically Generated Audio-Visual-Text Triplets from Videos](https://mlsanity.com/paper/2507.11967/) (2025)

## Abstract (arXiv)

Audio-visual captioning aims to generate holistic scene descriptions by jointly modeling sound and vision. While recent methods have improved performance through sophisticated modality fusion, it remains unclear to what extent the two modalities are complementary in current audio-visual captioning models and how robust these models are when one modality is degraded. We address these questions by conducting systematic modality robustness tests on LAVCap, a state-of-the-art audio-visual captioning model, in which we selectively suppress or corrupt the audio or visual streams to quantify sensitivity and complementarity. The analysis reveals a pronounced bias toward the audio stream in LAVCap. To evaluate how balanced audio-visual captioning models are in their use of both modalities, we augment AudioCaps with textual annotations that jointly describe the audio and visual streams, yielding the AudioVisualCaps dataset. In our experiments, we report LAVCap baseline results on AudioVisualCaps. We also evaluate the model under modality robustness tests on AudioVisualCaps and the results indicate that LAVCap trained on AudioVisualCaps exhibits less modality bias than when trained on AudioCaps.