# Frame Sampling Strategies Matter: A Benchmark for small vision language models

arXiv: https://arxiv.org/abs/2509.14769 · HTML: https://mlsanity.com/paper/2509.14769/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025)
- [SmolVLM: Redefining small and efficient multimodal models](https://mlsanity.com/paper/2504.05299/) (2025)
- [Ovis: Structural Embedding Alignment for Multimodal Large Language Model](https://mlsanity.com/paper/2405.20797/) (2024)
- [Frame-Voyager: Learning to Query Frames for Video Large Language Models](https://mlsanity.com/paper/2410.03226/) (2024)
- [BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding](https://mlsanity.com/paper/2503.21483/) (2025)
- [CSTA: CNN-based Spatiotemporal Attention for Video Summarization](https://mlsanity.com/paper/2405.11905/) (2024)
- [Hierarchical Boundary-Aware Neural Encoder for Video Captioning](https://mlsanity.com/paper/1611.09312/) (2016)
- [A Survey on Efficient Vision-Language Models](https://mlsanity.com/paper/2504.09724/) (2025)
- [MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understanding](https://mlsanity.com/paper/2502.03183/) (2025)
- [Small Vision-Language Models: A Survey on Compact Architectures and Techniques](https://mlsanity.com/paper/2503.10665/) (2025)

## Built on by (most cited first, 6 of 6)

- [PEEK: Picking Essential frames via Efficient Knowledge distillation](https://mlsanity.com/paper/2605.31029/) (2026)
- [Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs](https://mlsanity.com/paper/2608.07014/) (2026)
- [From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics](https://mlsanity.com/paper/2603.26772/) (2026)
- [MetaphorVU: Towards Metaphorical Video Understanding](https://mlsanity.com/paper/2605.25461/) (2026)
- [Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs](https://mlsanity.com/paper/2609.10355/) (2026)
- [Stable Scores, Unstable Answers: Frame Phase and Option Order in Video Multiple-Choice Evaluation](https://mlsanity.com/paper/2610.08649/) (2026)

## Abstract (arXiv)

Comparing vision language models on videos is particularly complex, as the performances is jointly determined by the model's visual representation capacity and the frame-sampling strategy used to construct the input. Current video benchmarks are suspected to suffer from substantial frame-sampling bias, as models are evaluated with different frame selection strategies. In this work, we propose the first frame-accurate benchmark of state-of-the-art small VLMs for video question-answering, evaluated under controlled frame-sampling strategies. Our results confirm the suspected bias and highlight both data-specific and task-specific behaviors of SVLMs under different frame-sampling techniques. By open-sourcing our benchmarking code, we provide the community with a reproducible and unbiased protocol for evaluating video VLMs and emphasize the need for standardized frame-sampling strategies tailored to each benchmarking dataset in future research.