# Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition arXiv: https://arxiv.org/abs/2307.02909 ยท HTML: https://mlsanity.com/paper/2307.02909/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Builds on 46 of its 106 references that are in the map. - First posted on arXiv 6 Jul 2023. ## Builds on (most cited first, 30 of 46) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015) - [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018) - [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018) - [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018) - [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016) - [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021) - [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020) - [The Conversation: Deep Audio-Visual Speech Enhancement](https://mlsanity.com/paper/1804.04121/) (2018) - [An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation](https://mlsanity.com/paper/2008.09586/) (2020) - [Robust Self-Supervised Audio-Visual Speech Recognition](https://mlsanity.com/paper/2201.01763/) (2022) - [End-to-end Audiovisual Speech Recognition](https://mlsanity.com/paper/1802.06424/) (2018) - [Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture](https://mlsanity.com/paper/1810.00108/) (2018) - [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019) - [Deep Multimodal Learning for Audio-Visual Speech Recognition](https://mlsanity.com/paper/1501.05396/) (2015) - [Far-Field Automatic Speech Recognition](https://mlsanity.com/paper/2009.09395/) (2020) - [Time Domain Audio Visual Speech Separation](https://mlsanity.com/paper/1904.03760/) (2019) - [Deep Lip Reading: a comparison of models and an online application](https://mlsanity.com/paper/1806.06053/) (2018) - [FaceFilter: Audio-visual speech separation using still images](https://mlsanity.com/paper/2005.07074/) (2020) - [Recognizing Overlapped Speech in Meetings: A Multichannel Separation Approach Using Neural Networks](https://mlsanity.com/paper/1810.03655/) (2018) - [Multi-modal Multi-channel Target Speech Separation](https://mlsanity.com/paper/2003.07032/) (2020) - [Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR](https://mlsanity.com/paper/2006.02786/) (2020) - [Modality Attention for End-to-End Audio-visual Speech Recognition](https://mlsanity.com/paper/1811.05250/) (2018) - [Audio-Visual Speech Separation and Dereverberation with a Two-Stage Multimodal Network](https://mlsanity.com/paper/1909.07352/) (2019) - [Uformer: A Unet based dilated complex & real dual-path conformer network for simultaneous speech enhancement and dereverberation](https://mlsanity.com/paper/2111.06015/) (2021) - [Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environments](https://mlsanity.com/paper/1811.02480/) (2018) - [Enhancing End-to-End Multi-channel Speech Separation via Spatial Feature Learning](https://mlsanity.com/paper/2003.03927/) (2020) - [Multimodal Attention Fusion for Target Speaker Extraction](https://mlsanity.com/paper/2102.01326/) (2021) ## Built on by (most cited first, 4 of 4) - [RIR-SF: Room Impulse Response Based Spatial Feature for Target Speech Recognition in Multi-Channel Multi-Speaker Scenarios](https://mlsanity.com/paper/2311.00146/) (2023) - [VioPose: Violin Performance 4D Pose Estimation by Hierarchical Audiovisual Inference](https://mlsanity.com/paper/2411.13607/) (2024) - [Supporting SENCOTEN Language Documentation Efforts with Automatic Speech Recognition](https://mlsanity.com/paper/2507.10827/) (2025) - [Visual-Informed Speech Enhancement Using Attention-Based Beamforming](https://mlsanity.com/paper/2603.05270/) (2026) ## Abstract (arXiv) Accurate recognition of cocktail party speech containing overlapping speakers, noise and reverberation remains a highly challenging task to date. Motivated by the invariance of visual modality to acoustic signal corruption, an audio-visual multi-channel speech separation, dereverberation and recognition approach featuring a full incorporation of visual information into all system components is proposed in this paper. The efficacy of the video input is consistently demonstrated in mask-based MVDR speech separation, DNN-WPE or spectral mapping (SpecM) based speech dereverberation front-end and Conformer ASR back-end. Audio-visual integrated front-end architectures performing speech separation and dereverberation in a pipelined or joint fashion via mask-based WPD are investigated. The error cost mismatch between the speech enhancement front-end and ASR back-end components is minimized by end-to-end jointly fine-tuning using either the ASR cost function alone, or its interpolation with the speech enhancement loss. Experiments were conducted on the mixture overlapped and reverberant speech data constructed using simulation or replay of the Oxford LRS2 dataset. The proposed audio-visual multi-channel speech separation, dereverberation and recognition systems consistently outperformed the comparable audio-only baseline by 9.1% and 6.2% absolute (41.7% and 36.0% relative) word error rate (WER) reductions. Consistent speech enhancement improvements were also obtained on PESQ, STOI and SRMR scores.