mlsanityData to 7 Oct 2026

Unsupervised Audio-Visual Lecture Segmentation

arXiv 2210.16644 ↗First posted 29 Oct 2022cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Over the last decade, online lecture videos have become increasingly popular and have experienced a meteoric rise during the pandemic. However, video-language research has primarily focused on instructional videos or movies, and tools to help students navigate the growing online lectures are lacking. Our first contribution is to facilitate research in the educational domain, by introducing AVLectures, a large-scale dataset consisting of 86 courses with over 2,350 lectures covering various STEM subjects. Each course contains video lectures, transcripts, OCR outputs for lecture frames, and optionally lecture notes, slides, assignments, and related educational content that can inspire a variety of tasks. Our second contribution is introducing video lecture segmentation that splits lectures into bite-sized topics that show promise in improving learner engagement. We formulate lecture segmentation as an unsupervised task that leverages visual, textual, and OCR cues from the lecture, while clip representations are fine-tuned on a pretext self-supervised task of matching the narration with the temporally aligned visual content. We use these representations to generate segments using a temporally consistent 1-nearest neighbor algorithm, TW-FINCH. We evaluate our method on 15 courses and compare it against various visual and textual baselines, outperforming all of them. Our comprehensive ablation studies also identify the key factors driving the success of our approach.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 29 Oct 2022.

Builds on

36 of 57 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  6. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  7. 2017Localizing Moments in Video with Natural Languagecited by 660
  8. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  9. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  10. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  11. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  12. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
24 more
  1. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  2. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  3. 2016Movie Descriptioncited by 180
  4. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  5. 2020Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherencecited by 121
  6. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  7. 2020Multi-modal Dense Video Captioningcited by 113
  8. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  9. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  10. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  11. 2021CLIP-It! Language-Guided Video Summarizationcited by 99
  12. 2019Efficient Parameter-free Clustering Using First Neighbor Relationscited by 96
  13. 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
  14. 2020A Local-to-Global Approach to Multi-modal Movie Scene Segmentationcited by 60
  15. 2021Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentationcited by 58
  16. 2015Video Paragraph Captioning Using Hierarchical Recurrent Neural Networkscited by 54
  17. 2019Weakly Supervised Energy-Based Learning for Action Segmentationcited by 53
  18. 2018Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interactioncited by 40
  19. 2018A Perceptual Prediction Framework for Self Supervised Event Segmentationcited by 40
  20. 2015Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioningcited by 30
  21. 2020Movie Summarization via Sparse Graph Constructioncited by 22
  22. 2017TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentationcited by 14
  23. 2021PEEK: A Large Dataset of Learner Engagement with Educational Videoscited by 9
  24. 2020VLEngagement: A Dataset of Scientific Video Lectures for Evaluating Population-based Engagementcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Attend to what I say: Highlighting relevant content on slidescited by 1
  2. 2026Reading Between the Waves: Robust Topic Segmentation Using Inter-Sentence Audio Featurescited by 1
  3. 2025Prompt2LVideos: Exploring Prompts for Understanding Long-Form Multimodal Videoscited by 0