Multimodal Abstractive Summarization of Instructional Videos with Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs
Abstract (arXiv)
Multimodal video summarization requires visual features that align semantically with language generation. Traditional approaches rely on CNN features trained for object classification, which represent visual concepts as discrete categories not aligned with natural language. We propose ClipSum, a framework that leverages frozen CLIP vision-language features with explicit temporal modeling and dimension-adaptive fusion for instructional video summarization. CLIP's contrastive pre-training on 400M image-text pairs yields visual features semantically aligned with the linguistic concepts that text decoders generate, bridging the vision-language gap at the representation level. On YouCook2, ClipSum achieves 33.0% ROUGE-1 versus 30.5% for ResNet-152 with 4x lower dimensionality (512 vs. 2048), demonstrating that semantic alignment matters more than feature capacity. Frozen CLIP (33.0%) surpasses fine-tuned CLIP (32.3%), showing that preserving pre-trained alignment is more valuable than task-specific adaptation. https://github.com/aqeeelmirza/clipsum
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 25 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
17 of 25 references.
References · 17
- 2015Deep Residual Learning for Image Recognitioncited by 51,950
- 2014Adam: A Method for Stochastic Optimizationcited by 37,093
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,616
- 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
- 2016Video Summarization with Long Short-term Memorycited by 142
- 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
5 more
- 2017Attention-Based Multimodal Fusion for Video Descriptioncited by 75
- 2019Multimodal Abstractive Summarization for How2 Videoscited by 41
- 2017Attention Strategies for Multi-Source Sequence-to-Sequence Learningcited by 32
- 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32
- 2016Video Summarization using Deep Semantic Featurescited by 15
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Realizing Video Summarization from the Path of Language-based Semantic Understanding
- 2025Minimal Clips, Maximum Salience: Long Video Summarization via Key Moment Extraction
- 2025Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structure
- 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions
- 2024Multimodal Language Models for Domain-Specific Procedural Video Summarization
- 2025VTD-CLIP: Video-to-Text Discretization via Prompting CLIP
- 2026Less is More: Label-Guided Efficient Summarization of Procedural and Instructional Videos
- 2026A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs