# The NPU-ASLP-LiAuto System Description for Visual Speech Recognition in CNVSRC 2023

arXiv: https://arxiv.org/abs/2401.06788 · HTML: https://mlsanity.com/paper/2401.06788/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 5 references that are in the map.
- First posted on arXiv 7 Jan 2024.

## Builds on (most cited first, 3 of 3)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding](https://mlsanity.com/paper/2207.02971/) (2022)

## Built on by (most cited first, 3 of 3)

- [Enhancing Lip Reading with Multi-Scale Video and Multi-Encoder](https://mlsanity.com/paper/2404.05466/) (2024)
- [CNVSRC 2024: The Second Chinese Continuous Visual Speech Recognition Challenge](https://mlsanity.com/paper/2506.02010/) (2025)
- [The NPU-ASLP System Description for Visual Speech Recognition in CNVSRC 2024](https://mlsanity.com/paper/2408.02369/) (2024)

## Abstract (arXiv)

This paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP-LiAuto (Team 237) in the first Chinese Continuous Visual Speech Recognition Challenge (CNVSRC) 2023, engaging in the fixed and open tracks of Single-Speaker VSR Task, and the open track of Multi-Speaker VSR Task. In terms of data processing, we leverage the lip motion extractor from the baseline1 to produce multi-scale video data. Besides, various augmentation techniques are applied during training, encompassing speed perturbation, random rotation, horizontal flipping, and color transformation. The VSR model adopts an end-to-end architecture with joint CTC/attention loss, comprising a ResNet3D visual frontend, an E-Branchformer encoder, and a Transformer decoder. Experiments show that our system achieves 34.76% CER for the Single-Speaker Task and 41.06% CER for the Multi-Speaker Task after multi-system fusion, ranking first place in all three tracks we participate.