# Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides

arXiv: https://arxiv.org/abs/2504.15066 · HTML: https://mlsanity.com/paper/2504.15066/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 21 Apr 2025.

## Builds on (most cited first, 20 of 20)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization](https://mlsanity.com/paper/2411.10442/) (2024)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://mlsanity.com/paper/2303.14307/) (2023)
- [Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture](https://mlsanity.com/paper/1810.00108/) (2018)
- [LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild](https://mlsanity.com/paper/1810.06990/) (2018)
- [Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization](https://mlsanity.com/paper/2305.11095/) (2023)
- [Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation](https://mlsanity.com/paper/2406.10082/) (2024)
- [A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Reading](https://mlsanity.com/paper/1908.04917/) (2019)
- [SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpus](https://mlsanity.com/paper/2309.05396/) (2023)
- [AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models](https://mlsanity.com/paper/2309.10787/) (2023)
- [AVATAR: Unconstrained Audiovisual Speech Recognition](https://mlsanity.com/paper/2206.07684/) (2022)
- [SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2401.09759/) (2024)
- [LCB-net: Long-Context Biasing for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2401.06390/) (2024)
- [CN-Celeb-AV: A Multi-Genre Audio-Visual Dataset for Person Recognition](https://mlsanity.com/paper/2305.16049/) (2023)
- [Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?](https://mlsanity.com/paper/2409.09221/) (2024)

## Built on by (most cited first, 5 of 5)

- [When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapse](https://mlsanity.com/paper/2603.22915/) (2026)
- [Unifying Speech Editing Detection and Content Localization via Prior-Enhanced Audio LLMs](https://mlsanity.com/paper/2601.21463/) (2026)
- [Index-MSR: A high-efficiency multimodal fusion framework for speech recognition](https://mlsanity.com/paper/2509.22744/) (2025)
- [Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2609.10394/) (2026)
- [ScreenASR-Bench: A Tiered Benchmark for Speech Recognition with Visual Context](https://mlsanity.com/paper/2610.16260/) (2026)

## Abstract (arXiv)

Incorporating visual modalities to assist Automatic Speech Recognition (ASR) tasks has led to significant improvements. However, existing Audio-Visual Speech Recognition (AVSR) datasets and methods typically rely solely on lip-reading information or speaking contextual video, neglecting the potential of combining these different valuable visual cues within the speaking context. In this paper, we release a multimodal Chinese AVSR dataset, Chinese-LiPS, comprising 100 hours of speech, video, and corresponding manual transcription, with the visual modality encompassing both lip-reading information and the presentation slides used by the speaker. Based on Chinese-LiPS, we develop a simple yet effective pipeline, LiPS-AVSR, which leverages both lip-reading and presentation slide information as visual modalities for AVSR tasks. Experiments show that lip-reading and presentation slide information improve ASR performance by approximately 8% and 25%, respectively, with a combined performance improvement of about 35%. The dataset is available at https://kiri0824.github.io/Chinese-LiPS/