# The NPU-ASLP System for Audio-Visual Speech Recognition in MISP 2022 Challenge

arXiv: https://arxiv.org/abs/2303.06341 · HTML: https://mlsanity.com/paper/2303.06341/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2023.

## Built on by (most cited first, 5 of 5)

- [A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2403.04245/) (2024)
- [MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition](https://mlsanity.com/paper/2401.03424/) (2024)
- [CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition](https://mlsanity.com/paper/2412.12760/) (2024)
- [ICMC-ASR: The ICASSP 2024 In-Car Multi-Channel Automatic Speech Recognition Challenge](https://mlsanity.com/paper/2401.03473/) (2024)
- [CabinSep: IR-Augmented Mask-Based MVDR for Real-Time In-Car Speech Separation with Distributed Heterogeneous Arrays](https://mlsanity.com/paper/2509.01399/) (2025)

## Abstract (arXiv)

This paper describes our NPU-ASLP system for the Audio-Visual Diarization and Recognition (AVDR) task in the Multi-modal Information based Speech Processing (MISP) 2022 Challenge. Specifically, the weighted prediction error (WPE) and guided source separation (GSS) techniques are used to reduce reverberation and generate clean signals for each single speaker first. Then, we explore the effectiveness of Branchformer and E-Branchformer based ASR systems. To better make use of the visual modality, a cross-attention based multi-modal fusion module is proposed, which explicitly learns the contextual relationship between different modalities. Experiments show that our system achieves a concatenated minimum-permutation character error rate (cpCER) of 28.13% and 31.21% on the Dev and Eval set, and obtains second place in the challenge.