mlsanityData to 7 Oct 2026

Zero-Shot Video Question Answering via Frozen Bidirectional Language Models

arXiv 2206.08155 ↗First posted 16 Jun 2022cs.CV cs.CL cs.LGcited by 229

Not on the map. Why?

Cited by 66 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle this problem, recent methods consider zero-shot settings with no manual annotation of visual question-answer. In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data to multi-modal inputs. In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA. In particular, (i) we combine visual inputs with the frozen BiLM using light trainable modules, (ii) we train such modules using Web-scraped multi-modal data, and finally (iii) we perform zero-shot VideoQA inference through masked language modeling, where the masked text is the answer to a given question. Our proposed approach, FrozenBiLM, outperforms the state of the art in zero-shot VideoQA by a significant margin on a variety of datasets, including LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA, TGIF-FrameQA, How2QA and TVQA. It also demonstrates competitive performance in the few-shot and fully-supervised setting. Our code and models are publicly available at https://github.com/antoyang/FrozenBiLM.

Key facts. Cited by 229 papers in the map (data through 7 Oct 2026). Builds on 105 of its 140 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

105 of 140 references.

References · 105

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2016Layer Normalizationcited by 5,579
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  13. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  14. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  15. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  16. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  17. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  18. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  19. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  20. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  21. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  22. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  23. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  24. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  25. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  26. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  27. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  28. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  29. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  30. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  31. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  32. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  33. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  34. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  35. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  36. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  37. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  38. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 69; 2024: 95; 2025: 49; 2026: 9.

229 papers.

Sort:

By area:

  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  5. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  6. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  7. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  8. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  9. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  10. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  11. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  12. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
38 more
  1. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  2. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  3. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  4. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  5. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  6. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  7. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  8. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  9. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  10. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  11. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  12. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  13. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  14. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  15. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  16. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  17. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  18. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  19. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 116
  20. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  21. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  22. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  23. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  24. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  25. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  26. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  27. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  28. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  29. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  30. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  31. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  32. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 83
  33. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  34. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 78
  35. 2023UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Webcited by 77
  36. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  37. 2023Audio-Visual LLM for Video Understandingcited by 76
  38. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74

50 most cited of 229 All 229 (JSON)