mlsanityData to 14 Oct 2026

Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides

arXiv 2504.15066 ↗First posted 21 Apr 2025cs.MM cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Incorporating visual modalities to assist Automatic Speech Recognition (ASR) tasks has led to significant improvements. However, existing Audio-Visual Speech Recognition (AVSR) datasets and methods typically rely solely on lip-reading information or speaking contextual video, neglecting the potential of combining these different valuable visual cues within the speaking context. In this paper, we release a multimodal Chinese AVSR dataset, Chinese-LiPS, comprising 100 hours of speech, video, and corresponding manual transcription, with the visual modality encompassing both lip-reading information and the presentation slides used by the speaker. Based on Chinese-LiPS, we develop a simple yet effective pipeline, LiPS-AVSR, which leverages both lip-reading and presentation slide information as visual modalities for AVSR tasks. Experiments show that lip-reading and presentation slide information improve ASR performance by approximately 8% and 25%, respectively, with a combined performance improvement of about 35%. The dataset is available at https://kiri0824.github.io/Chinese-LiPS/

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 20 of its 25 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

20 of 25 references.

References · 20

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2018Deep Audio-Visual Speech Recognitioncited by 322
  3. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  4. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  5. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  6. 2016Lip Reading Sentences in the Wildcited by 228
  7. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
  8. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  9. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  10. 2018LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wildcited by 51
  11. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  12. 2024Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translationcited by 36
8 more
  1. 2019A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Readingcited by 25
  2. 2023SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpuscited by 19
  3. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  4. 2022AVATAR: Unconstrained Audiovisual Speech Recognitioncited by 13
  5. 2024SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognitioncited by 7
  6. 2024LCB-net: Long-Context Biasing for Audio-Visual Speech Recognitioncited by 6
  7. 2023CN-Celeb-AV: A Multi-Genre Audio-Visual Dataset for Person Recognitioncited by 4
  8. 2024Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapsecited by 2
  2. 2026Unifying Speech Editing Detection and Content Localization via Prior-Enhanced Audio LLMscited by 1
  3. 2025Index-MSR: A high-efficiency multimodal fusion framework for speech recognitioncited by 0
  4. 2026Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognitioncited by 0
  5. 2026ScreenASR-Bench: A Tiered Benchmark for Speech Recognition with Visual Contextcited by 0