# Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Highlights in Videos

arXiv: https://arxiv.org/abs/2601.16333 · HTML: https://mlsanity.com/paper/2601.16333/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 56 references that are in the map.
- First posted on arXiv 22 Jan 2026.

## Builds on (most cited first, 29 of 29)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action](https://mlsanity.com/paper/2312.17172/) (2023)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos](https://mlsanity.com/paper/2011.13367/) (2020)
- [Trusted Multi-View Classification](https://mlsanity.com/paper/2102.02051/) (2021)
- [Describing Videos by Exploiting Temporal Structure](https://mlsanity.com/paper/1502.08029/) (2015)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [Probing Visual Language Priors in VLMs](https://mlsanity.com/paper/2501.00569/) (2024)
- [MatchTime: Towards Automatic Soccer Game Commentary Generation](https://mlsanity.com/paper/2406.18530/) (2024)
- [A Closer Look at Multimodal Representation Collapse](https://mlsanity.com/paper/2505.22483/) (2025)
- [Visual Question Answering on Image Sets](https://mlsanity.com/paper/2008.11976/) (2020)
- [ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models](https://mlsanity.com/paper/2311.07022/) (2023)
- [Adaptive Video Highlight Detection by Learning from User History](https://mlsanity.com/paper/2007.09598/) (2020)
- [Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2410.16983/) (2024)
- [Vision-Language Models Struggle to Align Entities across Modalities](https://mlsanity.com/paper/2503.03854/) (2025)
- [Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning](https://mlsanity.com/paper/2405.16869/) (2024)
- [TimeSoccer: An End-to-End Multimodal Large Language Model for Soccer Commentary Generation](https://mlsanity.com/paper/2504.17365/) (2025)
- [Leveraging Contextual Cues for Generating Basketball Highlights](https://mlsanity.com/paper/1606.08955/) (2016)
- [GROOViST: A Metric for Grounding Objects in Visual Storytelling](https://mlsanity.com/paper/2310.17770/) (2023)
- [Unsupervised Video Highlight Detection by Learning from Audio and Visual Recurrence](https://mlsanity.com/paper/2407.13933/) (2024)
- [Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition](https://mlsanity.com/paper/2407.04559/) (2024)
- [Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Tasks](https://mlsanity.com/paper/2410.03062/) (2024)
- [Automated Detection of Sport Highlights from Audio and Video Sources](https://mlsanity.com/paper/2501.16100/) (2025)
- [SoccerHigh: A Benchmark Dataset for Automatic Soccer Video Summarization](https://mlsanity.com/paper/2509.01439/) (2025)
- [Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questions](https://mlsanity.com/paper/2502.13034/) (2025)

## Abstract (arXiv)

Foundation models are used for many real-world applications involving language understanding and generation from temporally-ordered multimodal events. In this work, we study the ability of models to identify the most important sub-events in a video, which is a fundamental prerequisite for summarizing, reasoning over, or narrating multimodal events. Specifically, we focus on football games and evaluate models on their ability to distinguish between important and non-important sub-events in a game. To this end, we construct a new dataset by leveraging publicly available football highlight reels containing the most important moments of a game according to professional experts, without any additional annotation costs. Using our dataset, we compare several state-of-the-art multimodal models and show that they fail to reliably capture contextual importance. Analyses of models beyond standard evaluation metrics reveal their tendency to rely on a single dominant modality and their ineffectiveness in synthesizing necessary information from multiple sources. Our findings underline the importance of modular architectures that dynamically adapt to multimodal heterogeneity and the need for complementary training procedures that maximize cross-modal synergy.