mlsanityData to 14 Oct 2026

MEGA: Multimodal Alignment Aggregation and Distillation For Cinematic Video Segmentation

arXiv 2308.11185 ↗First posted 22 Aug 2023cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Previous research has studied the task of segmenting cinematic videos into scenes and into narrative acts. However, these studies have overlooked the essential task of multimodal alignment and fusion for effectively and efficiently processing long-form videos (>60min). In this paper, we introduce Multimodal alignmEnt aGgregation and distillAtion (MEGA) for cinematic long-video segmentation. MEGA tackles the challenge by leveraging multiple media modalities. The method coarsely aligns inputs of variable lengths and different modalities with alignment positional encoding. To maintain temporal synchronization while reducing computation, we further introduce an enhanced bottleneck fusion layer which uses temporal alignment. Additionally, MEGA employs a novel contrastive loss to synchronize and transfer labels across modalities, enabling act segmentation from labeled synopsis sentences on video shots. Our experimental results show that MEGA outperforms state-of-the-art methods on MovieNet dataset for scene segmentation (with an Average Precision improvement of +1.19%) and on TRIPOD dataset for act segmentation (with a Total Agreement improvement of +5.51%)

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 22 Aug 2023.

Builds on

36 of 53 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  7. 2020Supervised Contrastive Learningcited by 2,829
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2018Self-Attention with Relative Position Representationscited by 1,120
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  12. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
24 more
  1. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  2. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  3. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  4. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  5. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  6. 2015Cross Modal Distillation for Supervision Transfercited by 175
  7. 2020TransNet V2: An effective deep network architecture for fast shot transition detectioncited by 171
  8. 2020Deep Multimodal Fusion by Channel Exchangingcited by 118
  9. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 100
  10. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 64
  11. 2020A Local-to-Global Approach to Multi-modal Movie Scene Segmentationcited by 60
  12. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 51
  13. 2021Shot Contrastive Self-Supervised Learning for Scene Boundary Detectioncited by 35
  14. 2019Movie Plot Analysis via Turning Point Identificationcited by 34
  15. 2019A Graph-Based Framework to Bridge Movies and Synopsescited by 31
  16. 2020Screenplay Summarization Using Latent Narrative Structurecited by 23
  17. 2020Movie Summarization via Sparse Graph Constructioncited by 22
  18. 2015A Deep Siamese Network for Scene Detection in Broadcast Videoscited by 21
  19. 2015Beyond Frontal Faces: Improving Person Recognition Using Multiple Cuescited by 20
  20. 2022Boundary-aware Self-supervised Learning for Video Scene Segmentationcited by 17
  21. 2021Finding the Right Moment: Human-Assisted Trailer Creation via Task Compositioncited by 12
  22. 2018Unifying Identification and Context Learning for Person Recognitioncited by 10
  23. 2022Scene Consistency Representation Learning for Video Scene Segmentationcited by 9
  24. 2022Learning-by-Narrating: Narrative Pre-Training for Zero-Shot Dialogue Comprehensioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMscited by 9
  2. 2025Scene-VLM: Multimodal Video Scene Segmentation via Vision-Language Modelscited by 2
  3. 2026Open Your Model's Eyes: Video and Context-Aware Multimodal Backchannel Predictioncited by 1
  4. 2026Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAGcited by 0