mlsanityData to 14 Oct 2026

M3AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset

arXiv 2403.14168 ↗First posted 21 Mar 2024cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Publishing open-source academic video recordings is an emergent and prevalent approach to sharing knowledge online. Such videos carry rich multimodal information including speech, the facial and body movements of the speakers, as well as the texts and pictures in the slides and possibly even the papers. Although multiple academic video datasets have been constructed and released, few of them support both multimodal content recognition and understanding tasks, which is partially due to the lack of high-quality human annotations. In this paper, we propose a novel multimodal, multigenre, and multipurpose audio-visual academic lecture dataset (M3AV), which has almost 367 hours of videos from five sources covering computer science, mathematics, and medical and biology topics. With high-quality human annotations of the slide text and spoken words, in particular high-valued name entities, the dataset can be used for multiple audio-visual recognition and understanding tasks. Evaluations performed on contextual speech recognition, speech synthesis, and slide and script generation tasks demonstrate that the diversity of M3AV makes it a challenging dataset.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 34 of its 63 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

34 of 63 references.

References · 34

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  11. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  12. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
22 more
  1. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  3. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  4. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  5. 2015Listen, Attend and Spellcited by 543
  6. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  7. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  8. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  9. 2018Deep context: end-to-end contextual speech recognitioncited by 137
  10. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 101
  11. 2020Deep Shallow Fusion for RNN-T Personalizationcited by 64
  12. 2023DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversioncited by 38
  13. 2020Conversational End-to-End TTS for Voice Agentcited by 37
  14. 2023A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speechcited by 22
  15. 2018What Should I Learn First: Introducing LectureBank for NLP Education and Prerequisite Chain Learningcited by 21
  16. 2019TalkSumm: A Dataset and Scalable Annotation Method for Scientific Paper Summarization Based on Conference Talkscited by 20
  17. 2023SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpuscited by 19
  18. 2021See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarizationcited by 18
  19. 2022Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognitioncited by 17
  20. 2022Minimising Biasing Word Errors for Contextual ASR with the Tree-Constrained Pointer Generatorcited by 11
  21. 2022Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slidescited by 10
  22. 2023Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generatorcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 24
  2. 2026OCR-Enhanced Multimodal ASR Can Read While Listeningcited by 1
  3. 2025What's the Best Way to Retrieve Slides? A Comparative Study of Multimodal, Caption-Based, and Hybrid Retrieval Techniquescited by 0
  4. 2026AD4EDU: Benchmarking University Lecture Audio Description across Accessibility Standards, Metrics, and Learner Preferencescited by 0