# Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs

arXiv: https://arxiv.org/abs/2410.10441 · HTML: https://mlsanity.com/paper/2410.10441/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [Zero-Shot Video Question Answering via Frozen Bidirectional Language Models](https://mlsanity.com/paper/2206.08155/) (2022)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models](https://mlsanity.com/paper/2407.15841/) (2024)
- [An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM](https://mlsanity.com/paper/2403.18406/) (2024)
- [Audio-Visual LLM for Video Understanding](https://mlsanity.com/paper/2312.06720/) (2023)
- [Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens](https://mlsanity.com/paper/2312.08870/) (2023)
- [FreeVA: Offline MLLM as Training-Free Video Assistant](https://mlsanity.com/paper/2405.07798/) (2024)
- [Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning](https://mlsanity.com/paper/2405.05615/) (2024)

## Built on by (most cited first, 17 of 17)

- [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025)
- [Kwai Keye-VL 1.5 Technical Report](https://mlsanity.com/paper/2509.01563/) (2025)
- [Kwai Keye-VL Technical Report](https://mlsanity.com/paper/2507.01949/) (2025)
- [A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models](https://mlsanity.com/paper/2508.01548/) (2025)
- [p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay](https://mlsanity.com/paper/2412.04449/) (2024)
- [DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World](https://mlsanity.com/paper/2506.24102/) (2025)
- [Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2505.24164/) (2025)
- [Dense360: Dense Understanding from Omnidirectional Panoramas](https://mlsanity.com/paper/2506.14471/) (2025)
- [Flow4Agent: Long-form Video Understanding via Motion Prior from Optical Flow](https://mlsanity.com/paper/2510.05836/) (2025)
- [CyberV: Cybernetics for Test-time Scaling in Video Understanding](https://mlsanity.com/paper/2506.07971/) (2025)
- [KTV: Keyframes and Key Tokens Selection for Efficient Training-Free Video LLMs](https://mlsanity.com/paper/2602.03615/) (2026)
- [Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models](https://mlsanity.com/paper/2603.25155/) (2026)
- [ENTER: Event Based Interpretable Reasoning for VideoQA](https://mlsanity.com/paper/2501.14194/) (2025)
- [LLaVA-MLB: Mitigating and Leveraging Attention Bias for Training-Free Video LLMs](https://mlsanity.com/paper/2503.11205/) (2025)
- [AVAM: Universal Training-free Adaptive Visual Anchoring Embedded into Multimodal Large Language Model for Multi-image Question Answering](https://mlsanity.com/paper/2508.17860/) (2025)
- [VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference](https://mlsanity.com/paper/2603.29494/) (2026)
- [Sparrow: Data-Efficient Video-LLM with Text-to-Image Augmentation](https://mlsanity.com/paper/2411.19951/) (2024)

## Abstract (arXiv)

Vision-language large models have achieved remarkable success in various multi-modal tasks, yet applying them to video understanding remains challenging due to the inherent complexity and computational demands of video data. While training-based video-LLMs deliver high performance, they often require substantial resources for training and inference. Conversely, training-free approaches offer a more efficient alternative by adapting pre-trained image-LLMs models for video tasks without additional training, but they face inference efficiency bottlenecks due to the large number of visual tokens generated from video frames. In this work, we present a novel prompt-guided visual perception framework (abbreviated as Free Video-LLM) for efficient inference of training-free video LLMs. The proposed framework decouples spatial-temporal dimension and performs temporal frame sampling and spatial RoI cropping respectively based on task-specific prompts. Our method effectively reduces the number of visual tokens while maintaining high performance across multiple video question-answering benchmarks. Extensive experiments demonstrate that our approach achieves competitive results with significantly fewer tokens, offering an optimal trade-off between accuracy and computational efficiency compared to state-of-the-art video LLMs. The code will be available at https://github.com/contrastive/FreeVideoLLM.