# SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition

arXiv: https://arxiv.org/abs/2401.09759 · HTML: https://mlsanity.com/paper/2401.09759/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition](https://mlsanity.com/paper/2007.09127/) (2020)
- [Large-Scale Visual Speech Recognition](https://mlsanity.com/paper/1807.05162/) (2018)
- [Jointly Learning Visual and Auditory Speech Representations from Raw Data](https://mlsanity.com/paper/2212.06246/) (2022)
- [Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization](https://mlsanity.com/paper/2305.11095/) (2023)
- [JTubeSpeech: corpus of Japanese speech collected from YouTube for speech recognition and speaker verification](https://mlsanity.com/paper/2112.09323/) (2021)
- [Svarah: Evaluating English ASR Systems on Indian Accents](https://mlsanity.com/paper/2305.15760/) (2023)

## Built on by (most cited first, 7 of 7)

- [Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides](https://mlsanity.com/paper/2504.15066/) (2025)
- [Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?](https://mlsanity.com/paper/2409.09221/) (2024)
- [MLLM-based Speech Recognition: When and How is Multimodality Beneficial?](https://mlsanity.com/paper/2507.19037/) (2025)
- [OCR-Enhanced Multimodal ASR Can Read While Listening](https://mlsanity.com/paper/2601.18393/) (2026)
- [Unifying Scientific Communication: Fine-Grained Correspondence Across Scientific Media](https://mlsanity.com/paper/2605.05831/) (2026)
- [The Spoken Wikipedia Presentation Corpus](https://mlsanity.com/paper/2609.21676/) (2026)
- [ScreenASR-Bench: A Tiered Benchmark for Speech Recognition with Visual Context](https://mlsanity.com/paper/2610.16260/) (2026)

## Abstract (arXiv)

Audio-visual speech recognition (AVSR) is a multimodal extension of automatic speech recognition (ASR), using video as a complement to audio. In AVSR, considerable efforts have been directed at datasets for facial features such as lip-readings, while they often fall short in evaluating the image comprehension capabilities in broader contexts. In this paper, we construct SlideAVSR, an AVSR dataset using scientific paper explanation videos. SlideAVSR provides a new benchmark where models transcribe speech utterances with texts on the slides on the presentation recordings. As technical terminologies that are frequent in paper explanations are notoriously challenging to transcribe without reference texts, our SlideAVSR dataset spotlights a new aspect of AVSR problems. As a simple yet effective baseline, we propose DocWhisper, an AVSR model that can refer to textual information from slides, and confirm its effectiveness on SlideAVSR.