# MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features

arXiv: https://arxiv.org/abs/2409.16765 · HTML: https://mlsanity.com/paper/2409.16765/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 17 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 3 of 3)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content](https://mlsanity.com/paper/2110.04406/) (2021)
- [Presenting and Evaluating the Impact of Experiential Learning in Computing Accessibility Education](https://mlsanity.com/paper/2002.06445/) (2020)

## Built on by (most cited first, 2 of 2)

- [Unifying Scientific Communication: Fine-Grained Correspondence Across Scientific Media](https://mlsanity.com/paper/2605.05831/) (2026)
- [The Spoken Wikipedia Presentation Corpus](https://mlsanity.com/paper/2609.21676/) (2026)

## Abstract (arXiv)

This paper presents a benchmark dataset for aligning lecture videos with corresponding slides and introduces a novel multimodal algorithm leveraging features from speech, text, and images. It achieves an average accuracy of 0.82 in comparison to SIFT (0.56) while being approximately 11 times faster. Using dynamic programming the algorithm tries to determine the optimal slide sequence. The results show that penalizing slide transitions increases accuracy. Features obtained via optical character recognition (OCR) contribute the most to a high matching accuracy, followed by image features. The findings highlight that audio transcripts alone provide valuable information for alignment and are beneficial if OCR data is lacking. Variations in matching accuracy across different lectures highlight the challenges associated with video quality and lecture style. The novel multimodal algorithm demonstrates robustness to some of these challenges, underscoring the potential of the approach.