# Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs

arXiv: https://arxiv.org/abs/2609.03820 · HTML: https://mlsanity.com/paper/2609.03820/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs](https://mlsanity.com/paper/2506.22139/) (2025)
- [MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMs](https://mlsanity.com/paper/2501.02885/) (2025)
- [AdaCodec: A Predictive Visual Code for Video MLLMs](https://mlsanity.com/paper/2606.02569/) (2026)
- [Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding](https://mlsanity.com/paper/2604.01002/) (2026)
- [Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors](https://mlsanity.com/paper/2607.15689/) (2026)
- [LDDR: Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMs](https://mlsanity.com/paper/2605.11477/) (2026)
- [ReQuest: Rethinking-based Question-Aware Frame Selection for Long-Form Video QA](https://mlsanity.com/paper/2607.01737/) (2026)

## Built on by (most cited first, 1 of 1)

- [Stable Scores, Unstable Answers: Frame Phase and Option Order in Video Multiple-Choice Evaluation](https://mlsanity.com/paper/2610.08649/) (2026)

## Abstract (arXiv)

Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.