# Multimodal Confidence Modeling in Audio-Visual Quality Assessment

arXiv: https://arxiv.org/abs/2605.01219 · HTML: https://mlsanity.com/paper/2605.01219/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 22 references that are in the map.
- First posted on arXiv 2 May 2026.

## Builds on (most cited first, 10 of 10)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis](https://mlsanity.com/paper/2412.15322/) (2024)
- [AVSegFormer: Audio-Visual Segmentation with Transformer](https://mlsanity.com/paper/2307.01146/) (2023)
- [SCOREQ: Speech Quality Assessment with Contrastive Regression](https://mlsanity.com/paper/2410.06675/) (2024)
- [UNQA: Unified No-Reference Quality Assessment for Audio, Image, Video, and Audio-Visual Content](https://mlsanity.com/paper/2407.19704/) (2024)
- [NAViDAd: A No-Reference Audio-Visual Quality Metric Based on a Deep Autoencoder](https://mlsanity.com/paper/2001.11406/) (2020)
- [Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification](https://mlsanity.com/paper/2408.14441/) (2024)
- [MVAD: A Multiple Visual Artifact Detector for Video Streaming](https://mlsanity.com/paper/2406.00212/) (2024)
- [Convolutions Need Registers Too: HVS-Inspired Dynamic Attention for Video Quality Assessment](https://mlsanity.com/paper/2601.11045/) (2026)

## Abstract (arXiv)

Audio-visual quality assessment (AVQA) is essential for streaming, teleconferencing, and immersive media. In realistic streaming scenarios, distortions are often asymmetric, where one modality may be severely degraded while the other remains clean. Still, most contemporary AVQA metrics treat audio and video as equally reliable, causing confidence-unaware fusion to emphasize unreliable signals. This paper proposes MCM-AVQA, a multimodal confidence-aware AVQA framework that explicitly estimates modality-specific confidence and injects it into a dedicated audio-visual mixer for cross-modal attention. The Audio-Visual Mixer utilizes frame-level, confidence-guided channel attention to gate fusion, modulating feature interaction between modalities so that high-confidence streams dominate while unreliable inputs are suppressed, preserving temporal degradation patterns. A multi-head visual confidence estimator turns frame-level artifact probabilities into temporally smoothed, clip-level visual confidence scores, while an audio confidence module derives confidence from speech-quality cues without requiring a clean reference. Experiments on multiple AVQA benchmarks show that MCM-AVQA, and specifically its confidence-guided Audio-Visual Mixer, improve correlation with human mean opinion scores and yield more interpretable behavior under real-world asymmetric audio-visual distortions.