mlsanityData to 14 Oct 2026

FreeVA: Offline MLLM as Training-Free Video Assistant

arXiv 2405.07798 ↗First posted 13 May 2024cs.CV cs.AIcited by 29

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This paper undertakes an empirical study to revisit the latest advancements in Multimodal Large Language Models (MLLMs): Video Assistant. This study, namely FreeVA, aims to extend existing image-based MLLM to the video domain in a training-free manner. The study provides an essential, yet must-know baseline, and reveals several surprising findings: 1) FreeVA, leveraging only offline image-based MLLM without additional training, excels in zero-shot video question-answering (e.g., MSVD-QA, ActivityNet-QA, and MSRVTT-QA), even surpassing state-of-the-art methods that involve video instruction tuning. 2) While mainstream video-based MLLMs typically initialize with an image-based MLLM (e.g., LLaVA) and then fine-tune using video instruction tuning, the study indicates that utilizing the widely adopted VideoInstruct-100K for video instruction tuning doesn't actually lead to better performance compared to not training at all. 3) The commonly used evaluation metrics in existing works are significantly influenced by changes in the GPT API version over time. If ignored, this could affect the fairness and uniformity of comparisons between different methods and impact the analysis and judgment of researchers in the field. The advancement of MLLMs is currently thriving, drawing numerous researchers into the field. We aim for this work to serve as a plug-and-play, simple yet effective baseline, encouraging the direct evaluation of existing MLLMs in video domain while also standardizing the field of video conversational models to a certain extent. Also, we encourage researchers to reconsider: Have current video MLLM methods truly acquired knowledge beyond image MLLM? Code is available at https://github.com/whwu95/FreeVA

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 43 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

36 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
21 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  6. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  8. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  9. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  11. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  12. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  13. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  14. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  15. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 71
  16. 2023UATVR: Uncertainty-Adaptive Text-Video Retrievalcited by 55
  17. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 49
  18. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41
  19. 2023GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?cited by 35
  20. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understandingcited by 18
  21. 2024GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognitioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 9; 2026: 5.

29 papers.

Sort:

By area:

  1. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  2. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  3. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  4. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
  5. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
  6. 2024PruneVid: Visual Token Pruning for Efficient Video Large Language Modelscited by 67
  7. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 58
  8. 2024Dense Connector for MLLMscited by 49
  9. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videoscited by 38
  10. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 32
  11. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  12. 2024Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervisioncited by 25
17 more
  1. 2024STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Trainingcited by 22
  2. 2025UniVid: The Open-Source Unified Video Modelcited by 19
  3. 2026OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoningcited by 19
  4. 2024Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMscited by 17
  5. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 14
  6. 2024Beyond Training: Dynamic Token Merging for Zero-Shot Video Understandingcited by 13
  7. 2024DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understandingcited by 10
  8. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Modelscited by 9
  9. 2025FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understandingcited by 6
  10. 2026KTV: Keyframes and Key Tokens Selection for Efficient Training-Free Video LLMscited by 6
  11. 2025ENTER: Event Based Interpretable Reasoning for VideoQAcited by 5
  12. 2025LLaVA-MLB: Mitigating and Leveraging Attention Bias for Training-Free Video LLMscited by 2
  13. 2025D-CoDe: Scaling Image-Pretrained VLMs to Video via Dynamic Compression and Question Decompositioncited by 2
  14. 2026Unified Spatio-Temporal Token Scoring for Efficient Video VLMscited by 1
  15. 2025Harnessing Object Grounding for Time-Sensitive Video Understandingcited by 0
  16. 2026Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportationcited by 0
  17. 2026MEDR: Query-Independent Frame Selection via Multi-Signal Event Modeling and Dynamic Rescoringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluation of MLLM-Agnostic Plug-and-Play Keyframe Selection Methods for Long Video Understanding
  2. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
  3. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
  4. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
  5. 2024A Survey on Benchmarks of Multimodal Large Language Models
  6. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
  7. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
  8. 2026VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation