# Uncertainty-Aware and Decoder-Aligned Learning for Video Summarization

arXiv: https://arxiv.org/abs/2605.09507 · HTML: https://mlsanity.com/paper/2605.09507/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 10 May 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 9 of 9)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [CLIP-It! Language-Guided Video Summarization](https://mlsanity.com/paper/2107.00650/) (2021)
- [Summarizing Videos with Attention](https://mlsanity.com/paper/1812.01969/) (2018)
- [Align and Attend: Multimodal Summarization with Dual Contrastive Losses](https://mlsanity.com/paper/2303.07284/) (2023)
- [CSTA: CNN-based Spatiotemporal Attention for Video Summarization](https://mlsanity.com/paper/2405.11905/) (2024)
- [Progressive Video Summarization via Multimodal Self-supervised Learning](https://mlsanity.com/paper/2201.02494/) (2022)
- [Query Twice: Dual Mixture Attention Meta Learning for Video Summarization](https://mlsanity.com/paper/2008.08360/) (2020)
- [SummDiff: Generative Modeling of Video Summarization with Diffusion](https://mlsanity.com/paper/2510.08458/) (2025)

## Abstract (arXiv)

Video summarization aims to produce a compact representation of a long video by selecting a subset of temporally important segments that best reflect human preferences. This task is inherently difficult due to strong annotation subjectivity and the reliance on discrete decoding procedures, such as temporal segmentation and knapsack-based selection, during evaluation. Most existing approaches either learn deterministic importance scores that overlook these characteristics or adopt complex generative models that increase training and inference cost. In this paper, we propose VASTSum, an uncertainty-aware and decoder-aligned learning framework for video summarization that addresses both challenges within a single-pass model. The proposed method predicts probabilistic frame-level importance scores using a variational formulation, enabling explicit modeling of uncertainty arising from multi-annotator supervision. To account for subjectivity, particularly under binary annotations, we employ a supervision strategy that encourages alignment with plausible human annotation modes rather than enforcing a single consensus target. Furthermore, we introduce a decoder-aligned regularization that promotes stability of knapsack-based summary selection, reducing sensitivity to small perturbations in predicted scores. We evaluate the proposed framework on the SumMe and TVSum benchmarks using standard rank-based metrics. Experimental results show consistent and competitive Kendall and Spearman correlations across multiple data splits, demonstrating improved robustness under annotation disagreement while maintaining efficient single-forward inference. These results indicate that explicitly modeling uncertainty and aligning learning objectives with the decoding stage provide a principled alternative to both deterministic and diffusion-based video summarization methods.