mlsanityData to 7 Oct 2026

Zero-Shot Video Question Answering with Procedural Programs

arXiv 2312.00937 ↗First posted 1 Dec 2023cs.CVcited by 37

Not on the map. Why?

Abstract (arXiv)

We propose to answer zero-shot questions about videos by generating short procedural programs that derive a final answer from solving a sequence of visual subtasks. We present Procedural Video Querying (ProViQ), which uses a large language model to generate such programs from an input question and an API of visual modules in the prompt, then executes them to obtain the output. Recent similar procedural approaches have proven successful for image question answering, but videos remain challenging: we provide ProViQ with modules intended for video understanding, allowing it to generalize to a wide variety of videos. This code generation framework additionally enables ProViQ to perform other video tasks in addition to question answering, such as multi-object tracking or basic video editing. ProViQ achieves state-of-the-art results on a diverse range of benchmarks, with improvements of up to 25% on short, long, open-ended, and multimodal video question-answering datasets. Our project page is at https://rccchoudhury.github.io/proviq2023.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 60 of its 65 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

60 of 65 references.

References · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  12. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  13. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  14. 2021ByteTrack: Multi-Object Tracking by Associating Every Detection Boxcited by 812
  15. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  16. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  17. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  18. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  19. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  20. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  21. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  22. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  23. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  24. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  25. 2015Neural Module Networkscited by 375
  26. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  27. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  28. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  29. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  30. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  31. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  32. 2022Learning Video Representations from Large Language Modelscited by 204
  33. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  34. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  35. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  36. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  37. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  38. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 15; 2025: 19; 2026: 2.

37 papers.

Sort:

By area:

  1. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  2. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  3. 2023Video Understanding with Large Language Models: A Surveycited by 203
  4. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  5. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  6. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  7. 2024Language Repository for Long Video Understandingcited by 55
  8. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 36
  9. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  10. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 30
  11. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  12. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 24
25 more
  1. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  2. 2025Agentic Keyframe Search for Video Question Answeringcited by 13
  3. 2025VTimeCoT: Thinking by Drawing for Video Temporal Grounding and Reasoningcited by 11
  4. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  5. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  6. 2024HCQA @ Ego4D EgoSchema Challenge 2024cited by 9
  7. 2025RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graphcited by 7
  8. 2025SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Modelscited by 7
  9. 2025NeuS-QA: Grounding Long-Form Video Understanding in Temporal Logic and Neuro-Symbolic Reasoningcited by 7
  10. 2025HCQA-1.5 @ Ego4D EgoSchema Challenge 2025cited by 6
  11. 2025MECD+: Unlocking Event-Level Causal Graph Discovery for Video Reasoningcited by 5
  12. 2025ENTER: Event Based Interpretable Reasoning for VideoQAcited by 5
  13. 2025ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validationcited by 5
  14. 2024Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experiencecited by 4
  15. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Modelscited by 4
  16. 2025MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understandingcited by 4
  17. 2025VideoPro: Adaptive Program Reasoning for Long Video Understandingcited by 3
  18. 2025Confidence-guided Refinement Reasoning for Zero-shot Question Answeringcited by 3
  19. 2025Infinite Video Understandingcited by 2
  20. 2025Test-time Scaling of LLMs: A Survey from A Subproblem Structure Perspectivecited by 2
  21. 2026RT-NeuS: Towards Real-Time Neuro-Symbolic Video Understanding via Adaptive Temporal Verificationcited by 2
  22. 2025Four Eyes Are Better Than Two: Harnessing the Collaborative Potential of Large Models via Differentiated Thinking and Complementary Ensemblescited by 1
  23. 2025VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understandingcited by 1
  24. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Surveycited by 0
  25. 2026One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognitioncited by 0