mlsanityData to 7 Oct 2026

Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners

arXiv 2205.10747 ↗First posted 22 May 2022cs.CV cs.AIcited by 117

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction. Existing few-shot video-language learners focus exclusively on the encoder, resulting in the absence of a video-to-text decoder to handle generative tasks. Video captioners have been pretrained on large-scale video-language datasets, but they rely heavily on finetuning and lack the ability to generate text for unseen tasks in a few-shot setting. We propose VidIL, a few-shot Video-language Learner via Image and Language models, which demonstrates strong performance on few-shot video-to-text tasks without the necessity of pretraining or finetuning on any video datasets. We use the image-language models to translate the video content into frame captions, object, attribute, and event phrases, and compose them into a temporal structure template. We then instruct a language model, with a prompt containing a few in-context examples, to generate a target output from the composed content. The flexibility of prompting allows the model to capture any form of text input, such as automatic speech recognition (ASR) transcripts. Our experiments demonstrate the power of language models in understanding videos on a wide variety of video-language tasks, including video captioning, video question answering, video caption retrieval, and video future event prediction. Especially, on video future event prediction, our few-shot model significantly outperforms state-of-the-art supervised models trained on large-scale video datasets. Code and resources are publicly available for research purposes at https://github.com/MikeWangWZHL/VidIL .

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 64 of its 76 references that are in the map. First posted on arXiv 22 May 2022.

Builds on

64 of 76 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  5. 2018SlowFast Networks for Video Recognitioncited by 1,862
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  13. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  14. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  15. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  17. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  18. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  19. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  20. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  21. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  22. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  23. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  24. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  25. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  26. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  27. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  28. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  29. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  30. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  31. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  32. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  33. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  34. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  35. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  36. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  37. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  38. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 46; 2024: 35; 2025: 24; 2026: 2.

117 papers.

Sort:

By area:

  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  2. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  3. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  4. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  5. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  6. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  7. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 194
  8. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  9. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  10. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  11. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  12. 2022Linearly Mapping from Image to Text Spacecited by 139
38 more
  1. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  3. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  4. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  5. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  6. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  7. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85
  8. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  9. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  10. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  11. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  12. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  13. 2023Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillingscited by 68
  14. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  15. 2024Language Repository for Long Video Understandingcited by 55
  16. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  17. 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
  18. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  19. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  20. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  21. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  22. 2023Modular Visual Question Answering via Code Generationcited by 47
  23. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  24. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  25. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  26. 2023InstructVid2Vid: Controllable Video Editing with Natural Language Instructionscited by 44
  27. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 44
  28. 2023Multimodal Procedural Planning via Dual Text-Image Promptingcited by 40
  29. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  30. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  31. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 37
  32. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37
  33. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  34. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29
  35. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  36. 2022Learning to Decompose Visual Features with Latent Textual Promptscited by 28
  37. 2023VicTR: Video-conditioned Text Representations for Activity Recognitioncited by 28
  38. 2023SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Modelscited by 28

50 most cited of 117 All 117 (JSON)