# Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders arXiv: https://arxiv.org/abs/2505.24158 ยท HTML: https://mlsanity.com/paper/2505.24158/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 74 of its 81 references that are in the map. - First posted on arXiv 30 May 2025. ## Placed in - Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/) ## Builds on (most cited first, 30 of 74) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022) - [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023) - [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023) - [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024) - [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023) - [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023) - [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015) - [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018) - [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024) - [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023) - [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024) - [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014) - [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023) - [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024) - [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014) - [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023) - [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024) - [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024) - [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021) ## Built on by (most cited first, 10 of 10) - [Video-EM: Event-Centric Episodic Memory for Long-Form Video Understanding](https://mlsanity.com/paper/2508.09486/) (2025) - [FrameOracle: Learning What to See and How Much to See in Videos](https://mlsanity.com/paper/2510.03584/) (2025) - [Event-Anchored Frame Selection for Effective Long-Video Understanding](https://mlsanity.com/paper/2603.00983/) (2026) - [Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs](https://mlsanity.com/paper/2510.00705/) (2025) - [Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping](https://mlsanity.com/paper/2605.10937/) (2026) - [One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding](https://mlsanity.com/paper/2608.05707/) (2026) - [Training-Free Temporal Abstraction for General Video Understanding](https://mlsanity.com/paper/2608.27929/) (2026) - [Mathematical Programming in Machine Learning and Artificial Intelligence: A Unified Taxonomy of Models and Applications](https://mlsanity.com/paper/2609.07254/) (2026) - [MetaSampling: Making Frame Samplers Efficient for Long-Video Question Answering](https://mlsanity.com/paper/2609.33998/) (2026) - [FORTE: Adaptive Scoring and Exact Keyframe Selection for Long-Video Question Answering](https://mlsanity.com/paper/2610.00573/) (2026) ## Abstract (arXiv) Employing Multimodal Large Language Models (MLLMs) for long video understanding remains a challenging problem due to the dilemma between the substantial number of video frames (i.e., visual tokens) versus the limited context length of language models. Traditional uniform sampling often leads to selection of irrelevant content, while post-training MLLMs on thousands of frames imposes a substantial computational burden. In this paper, we propose threading keyframes with narratives (Nar-KFC), a plug-and-play module to facilitate effective and efficient long video perception. Nar-KFC generally involves two collaborative steps. First, we formulate the keyframe selection process as an integer quadratic programming problem, jointly optimizing query-relevance and frame-diversity. To avoid its computational complexity, a customized greedy search strategy is designed as an efficient alternative. Second, to mitigate the temporal discontinuity caused by sparse keyframe sampling, we further introduce interleaved textual narratives generated from non-keyframes using off-the-shelf captioners. These narratives are inserted between keyframes based on their true temporal order, forming a coherent and compact representation. Nar-KFC thus serves as a temporal- and content-aware compression strategy that complements visual and textual modalities. Experimental results on multiple long-video benchmarks demonstrate that Nar-KFC significantly improves the performance of popular MLLMs. Code will be made publicly available.