# When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapse

arXiv: https://arxiv.org/abs/2603.22915 · HTML: https://mlsanity.com/paper/2603.22915/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://mlsanity.com/paper/2303.14307/) (2023)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring](https://mlsanity.com/paper/2303.08536/) (2023)
- [Large Language Models are Strong Audio-Visual Speech Recognition Learners](https://mlsanity.com/paper/2409.12319/) (2024)
- [DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement](https://mlsanity.com/paper/2305.08227/) (2023)
- [ASR Error Correction using Large Language Models](https://mlsanity.com/paper/2409.09554/) (2024)
- [Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition](https://mlsanity.com/paper/1809.01728/) (2018)
- [Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation](https://mlsanity.com/paper/2406.05983/) (2024)
- [A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2403.04245/) (2024)
- [Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMs](https://mlsanity.com/paper/2510.22603/) (2025)
- [Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach](https://mlsanity.com/paper/2505.14336/) (2025)
- [Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations](https://mlsanity.com/paper/2503.06273/) (2025)
- [Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer](https://mlsanity.com/paper/2405.12983/) (2024)
- [Investigating the Lombard Effect Influence on End-to-End Audio-Visual Speech Recognition](https://mlsanity.com/paper/1906.02112/) (2019)
- [Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech Recognition](https://mlsanity.com/paper/2008.02686/) (2020)
- [Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides](https://mlsanity.com/paper/2504.15066/) (2025)
- [mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition](https://mlsanity.com/paper/2502.01547/) (2025)
- [Towards Decoupling Frontend Enhancement and Backend Recognition in Monaural Robust ASR](https://mlsanity.com/paper/2403.06387/) (2024)
- [DCIM-AVSR : Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module](https://mlsanity.com/paper/2409.00481/) (2024)
- [Hourglass-AVSR: Down-Up Sampling-based Computational Efficiency Model for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2312.08850/) (2023)
- [Enhancing the Robustness of Contextual ASR to Varying Biasing Information Volumes Through Purified Semantic Correlation Joint Modeling](https://mlsanity.com/paper/2509.05908/) (2025)

## Built on by (most cited first, 2 of 2)

- [AVSRBench: A Multi-Condition AVSR Benchmark](https://mlsanity.com/paper/2609.10366/) (2026)
- [Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2609.10394/) (2026)

## Abstract (arXiv)

Audio-Visual Speech Recognition (AVSR) has achieved remarkable progress in offline conditions, yet its robustness in real-world video conferencing (VC) remains largely unexplored. This paper presents the first systematic evaluation of state-of-the-art AVSR models across mainstream VC platforms, revealing severe performance degradation caused by transmission distortions and spontaneous human hyper-expression. To address this gap, we construct MLD-VC, the first multimodal dataset tailored for VC, comprising 31 speakers, 22.79 hours of audio-visual data, and explicit use of the Lombard effect to enhance human hyper-expression. Through comprehensive analysis, we find that speech enhancement algorithms are the primary source of distribution shift, which alters the first and second formants of audio. Interestingly, we find that the distribution shift induced by the Lombard effect closely resembles that introduced by speech enhancement, which explains why models trained on Lombard data exhibit greater robustness in VC. Fine-tuning AVSR models on MLD-VC mitigates this issue, achieving an average 17.5% reduction in CER across several VC platforms. Our findings and dataset provide a foundation for developing more robust and generalizable AVSR systems in real-world video conferencing. MLD-VC is available at https://huggingface.co/datasets/nccm2p2/MLD-VC.