# BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding

arXiv: https://arxiv.org/abs/2503.21483 · HTML: https://mlsanity.com/paper/2503.21483/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 35 of its 40 references that are in the map.
- First posted on arXiv 27 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models](https://mlsanity.com/paper/2407.12772/) (2024)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding](https://mlsanity.com/paper/2409.14485/) (2024)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models](https://mlsanity.com/paper/2407.15841/) (2024)
- [Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning](https://mlsanity.com/paper/2402.11435/) (2024)
- [MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens](https://mlsanity.com/paper/2404.03413/) (2024)
- [VideoLLM: Modeling Video Sequence with Large Language Models](https://mlsanity.com/paper/2305.13292/) (2023)
- [An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM](https://mlsanity.com/paper/2403.18406/) (2024)
- [MM-VID: Advancing Video Understanding with GPT-4V(ision)](https://mlsanity.com/paper/2310.19773/) (2023)
- [Frame-Voyager: Learning to Query Frames for Video Large Language Models](https://mlsanity.com/paper/2410.03226/) (2024)

## Built on by (most cited first, 30 of 75)

- [FOCUS: Efficient Keyframe Selection for Long Video Understanding](https://mlsanity.com/paper/2510.27280/) (2025)
- [StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding](https://mlsanity.com/paper/2508.01875/) (2025)
- [VideoITG: Multimodal Video Understanding with Instructed Temporal Grounding](https://mlsanity.com/paper/2507.13353/) (2025)
- [Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames](https://mlsanity.com/paper/2507.02001/) (2025)
- [FlexSelect: Flexible Token Selection for Efficient Long Video Understanding](https://mlsanity.com/paper/2506.00993/) (2025)
- [LensWalk: Agentic Video Understanding by Planning How You See in Videos](https://mlsanity.com/paper/2603.24558/) (2026)
- [ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding](https://mlsanity.com/paper/2506.01274/) (2025)
- [E-VRAG: Enhancing Long Video Understanding with Resource-Efficient Retrieval Augmented Generation](https://mlsanity.com/paper/2508.01546/) (2025)
- [Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding](https://mlsanity.com/paper/2512.04000/) (2025)
- [Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders](https://mlsanity.com/paper/2505.24158/) (2025)
- [Video-EM: Event-Centric Episodic Memory for Long-Form Video Understanding](https://mlsanity.com/paper/2508.09486/) (2025)
- [From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understanding](https://mlsanity.com/paper/2510.02262/) (2025)
- [Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism](https://mlsanity.com/paper/2603.29252/) (2026)
- [Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Search](https://mlsanity.com/paper/2601.13719/) (2026)
- [FrameOracle: Learning What to See and How Much to See in Videos](https://mlsanity.com/paper/2510.03584/) (2025)
- [Event-Anchored Frame Selection for Effective Long-Video Understanding](https://mlsanity.com/paper/2603.00983/) (2026)
- [Frame Sampling Strategies Matter: A Benchmark for small vision language models](https://mlsanity.com/paper/2509.14769/) (2025)
- [KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understanding](https://mlsanity.com/paper/2512.14017/) (2025)
- [MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding](https://mlsanity.com/paper/2602.22932/) (2026)
- [SeViCES: Unifying Semantic-Visual Evidence Consensus for Long Video Understanding](https://mlsanity.com/paper/2510.20622/) (2025)
- [CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management](https://mlsanity.com/paper/2603.19571/) (2026)
- [Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Sampling](https://mlsanity.com/paper/2606.24187/) (2026)
- [Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs](https://mlsanity.com/paper/2510.00705/) (2025)
- [Venus: An Efficient Edge Memory-and-Retrieval System for VLM-based Online Video Understanding](https://mlsanity.com/paper/2512.07344/) (2025)
- [HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering](https://mlsanity.com/paper/2512.14870/) (2025)
- [HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering](https://mlsanity.com/paper/2603.18558/) (2026)
- [Video-in-the-Loop: Span-Grounded Long Video QA with Interleaved Reasoning](https://mlsanity.com/paper/2510.04022/) (2025)
- [A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering](https://mlsanity.com/paper/2510.04428/) (2025)
- [QuPAINT: Physics-Aware Instruction Tuning Approach to Quantum Material Discovery](https://mlsanity.com/paper/2602.17478/) (2026)
- [ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting](https://mlsanity.com/paper/2603.23186/) (2026)

## Abstract (arXiv)

Large video-language models (VLMs) have demonstrated promising progress in various video understanding tasks. However, their effectiveness in long-form video analysis is constrained by limited context windows. Traditional approaches, such as uniform frame sampling, often inevitably allocate resources to irrelevant content, diminishing their effectiveness in real-world scenarios. In this paper, we introduce BOLT, a method to BOost Large VLMs without additional Training through a comprehensive study of frame selection strategies. First, to enable a more realistic evaluation of VLMs in long-form video understanding, we propose a multi-source retrieval evaluation setting. Our findings reveal that uniform sampling performs poorly in noisy contexts, underscoring the importance of selecting the right frames. Second, we explore several frame selection strategies based on query-frame similarity and analyze their effectiveness at inference time. Our results show that inverse transform sampling yields the most significant performance improvement, increasing accuracy on the Video-MME benchmark from 53.8% to 56.1% and MLVU benchmark from 58.9% to 63.4%. Our code is available at https://github.com/sming256/BOLT.