# CFSum: A Transformer-Based Multi-Modal Video Summarization Framework With Coarse-Fine Fusion

arXiv: https://arxiv.org/abs/2503.00364 · HTML: https://mlsanity.com/paper/2503.00364/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 1 Mar 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [CLIP-It! Language-Guided Video Summarization](https://mlsanity.com/paper/2107.00650/) (2021)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries](https://mlsanity.com/paper/2107.09609/) (2021)
- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [Less is More: Learning Highlight Detection from Video Duration](https://mlsanity.com/paper/1903.00859/) (2019)
- [TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2401.02309/) (2024)
- [MINI-Net: Multiple Instance Ranking Network for Video Highlight Detection](https://mlsanity.com/paper/2007.09833/) (2020)
- [MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer](https://mlsanity.com/paper/2305.00355/) (2023)
- [Cross-category Video Highlight Detection via Set-based Learning](https://mlsanity.com/paper/2108.11770/) (2021)
- [TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency](https://mlsanity.com/paper/2208.06773/) (2022)
- [R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding](https://mlsanity.com/paper/2404.00801/) (2024)
- [Summary Transfer: Exemplar-based Subset Selection for Video Summarization](https://mlsanity.com/paper/1603.03369/) (2016)
- [Probing Visual-Audio Representation for Video Highlight Detection via Hard-Pairs Guided Contrastive Learning](https://mlsanity.com/paper/2206.10157/) (2022)
- [Multimodal Frame-Scoring Transformer for Video Summarization](https://mlsanity.com/paper/2207.01814/) (2022)

## Built on by (most cited first, 3 of 3)

- [SoccerHigh: A Benchmark Dataset for Automatic Soccer Video Summarization](https://mlsanity.com/paper/2509.01439/) (2025)
- [TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization](https://mlsanity.com/paper/2603.01169/) (2026)
- [SD-MVSum: Script-Driven Multimodal Video Summarization Method and Datasets](https://mlsanity.com/paper/2510.05652/) (2025)

## Abstract (arXiv)

Video summarization, by selecting the most informative and/or user-relevant parts of original videos to create concise summary videos, has high research value and consumer demand in today's video proliferation era. Multi-modal video summarization that accomodates user input has become a research hotspot. However, current multi-modal video summarization methods suffer from two limitations. First, existing methods inadequately fuse information from different modalities and cannot effectively utilize modality-unique features. Second, most multi-modal methods focus on video and text modalities, neglecting the audio modality, despite the fact that audio information can be very useful in certain types of videos. In this paper we propose CFSum, a transformer-based multi-modal video summarization framework with coarse-fine fusion. CFSum exploits video, text, and audio modal features as input, and incorporates a two-stage transformer-based feature fusion framework to fully utilize modality-unique information. In the first stage, multi-modal features are fused simultaneously to perform initial coarse-grained feature fusion, then, in the second stage, video and audio features are explicitly attended with the text representation yielding more fine-grained information interaction. The CFSum architecture gives equal importance to each modality, ensuring that each modal feature interacts deeply with the other modalities. Our extensive comparative experiments against prior methods and ablation studies on various datasets confirm the effectiveness and superiority of CFSum.