mlsanityData to 7 Oct 2026

An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM

arXiv 2403.18406 ↗First posted 27 Mar 2024cs.CV cs.AI cs.CLcited by 97

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Stimulated by the sophisticated reasoning capabilities of recent Large Language Models (LLMs), a variety of strategies for bridging video modality have been devised. A prominent strategy involves Video Language Models (VideoLMs), which train a learnable interface with video data to connect advanced vision encoders with LLMs. Recently, an alternative strategy has surfaced, employing readily available foundation models, such as VideoLMs and LLMs, across multiple stages for modality bridging. In this study, we introduce a simple yet novel strategy where only a single Vision Language Model (VLM) is utilized. Our starting point is the plain insight that a video comprises a series of images, or frames, interwoven with temporal information. The essence of video comprehension lies in adeptly managing the temporal aspects along with the spatial details of each frame. Initially, we transform a video into a single composite image by arranging multiple frames in a grid layout. The resulting single image is termed as an image grid. This format, while maintaining the appearance of a solitary image, effectively retains temporal information within the grid structure. Therefore, the image grid approach enables direct application of a single high-performance VLM without necessitating any video-data training. Our extensive experimental analysis across ten zero-shot video question answering benchmarks, including five open-ended and five multiple-choice benchmarks, reveals that the proposed Image Grid Vision Language Model (IG-VLM) surpasses the existing methods in nine out of ten benchmarks.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 52 of its 59 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

52 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions20 Nov
  2. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
38 more
  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  13. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  14. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  15. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  16. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  17. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  18. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  19. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  20. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  21. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  22. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  23. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  24. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  25. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  26. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  27. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  28. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  29. 2023Pengi: An Audio Language Model for Audio Taskscited by 217
  30. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  31. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  32. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  33. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  34. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  35. 2023Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokenscited by 70
  36. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  37. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  38. 2023ViLA: Efficient Video-Language Alignment for Video Question Answeringcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 37; 2026: 25.

97 papers.

Sort:

By area:

  1. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  2. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  3. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  4. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  5. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  6. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  7. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  8. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  9. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  10. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  11. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  12. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96
38 more
  1. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  2. 2024Matryoshka Multimodal Modelscited by 82
  3. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  4. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 72
  5. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  6. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
  7. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  8. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  9. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  10. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  11. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  12. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  13. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  14. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  15. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  16. 2024Pathways on the Image Manifold: Image Editing via Video Generationcited by 27
  17. 2025CoS: Chain-of-Shot Prompting for Long Video Understandingcited by 27
  18. 2024LVD-2M: A Long-take Video Dataset with Temporally Dense Captionscited by 24
  19. 2025Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMscited by 19
  20. 2025UniVid: The Open-Source Unified Video Modelcited by 18
  21. 2024Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMscited by 17
  22. 2025VideoMultiAgents: A Multi-Agent Framework for Video Question Answeringcited by 16
  23. 2024Beyond Training: Dynamic Token Merging for Zero-Shot Video Understandingcited by 13
  24. 2025Agentic Keyframe Search for Video Question Answeringcited by 13
  25. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  26. 2024Question-Answering Dense Video Eventscited by 11
  27. 2025Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understandingcited by 11
  28. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  29. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  30. 2024Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Mannercited by 10
  31. 2025Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenderscited by 10
  32. 2026OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligencecited by 10
  33. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Modelscited by 9
  34. 2024Free2Guide: Training-Free Text-to-Video Alignment using Image LVLMcited by 9
  35. 2025TimeLogic: A Temporal Logic Benchmark for Video QAcited by 9
  36. 2025AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understandingcited by 9
  37. 2025Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansioncited by 8
  38. 2026Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understandingcited by 8

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
  2. 2025Bridging Vision Language Models and Symbolic Grounding for Video Question Answering
  3. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2025Frame Sampling Strategies Matter: A Benchmark for small vision language models
  6. 2024Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis
  7. 2024DARE: Diverse Visual Question Answering with Robustness Evaluation
  8. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts