mlsanityData to 7 Oct 2026

Prompting Visual-Language Models for Efficient Video Understanding

arXiv 2112.04478 ↗First posted 8 Dec 2021cs.CV cs.CLcited by 309

Not on the map. Why?

Cited by 70 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Image-based visual-language (I-VL) pre-training has shown great success for learning joint visual-textual representations from large-scale web data, revealing remarkable ability for zero-shot generalisation. This paper presents a simple but strong baseline to efficiently adapt the pre-trained I-VL model, and exploit its powerful ability for resource-hungry video understanding tasks, with minimal training. Specifically, we propose to optimise a few random vectors, termed as continuous prompt vectors, that convert video-related tasks into the same format as the pre-training objectives. In addition, to bridge the gap between static images and videos, temporal information is encoded with lightweight Transformers stacking on top of frame-wise visual features. Experimentally, we conduct extensive ablation studies to analyse the critical components. On 10 public benchmarks of action recognition, action localisation, and text-video retrieval, across closed-set, few-shot, and zero-shot scenarios, we achieve competitive or state-of-the-art performance to existing methods, despite optimising significantly fewer parameters.

Key facts. Cited by 309 papers in the map (data through 7 Oct 2026). Builds on 72 of its 108 references that are in the map. First posted on arXiv 8 Dec 2021.

Builds on

72 of 108 references.

References · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2017Non-local Neural Networkscited by 2,290
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2018SlowFast Networks for Video Recognitioncited by 1,862
  11. 2021ViViT: A Video Vision Transformercited by 1,610
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
38 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  4. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  5. 2021Video Swin Transformercited by 1,090
  6. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  8. 2016HyperNetworkscited by 1,003
  9. 2019How Can We Know What Language Models Know?cited by 953
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  12. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  13. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  14. 2017Localizing Moments in Video with Natural Languagecited by 660
  15. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  16. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  17. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  18. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  19. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  20. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  21. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  22. 2020Multi-modal Transformer for Video Retrievalcited by 395
  23. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  24. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 380
  25. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 327
  26. 2017Temporal Relational Reasoning in Videoscited by 324
  27. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  28. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  29. 2018BSN: Boundary Sensitive Network for Temporal Action Proposal Generationcited by 271
  30. 2017Temporal Action Detection with Structured Segment Networkscited by 265
  31. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 248
  32. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  33. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  34. 2018Rethinking the Faster R-CNN Architecture for Temporal Action Localizationcited by 194
  35. 2016Movie Descriptioncited by 180
  36. 2017R-C3D: Region Convolutional 3D Network for Temporal Activity Detectioncited by 161
  37. 2021Learning Salient Boundary Feature for Anchor-free Temporal Action Localizationcited by 156
  38. 2019Few-Shot Video Classification via Temporal Alignmentcited by 124

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 34; 2023: 111; 2024: 95; 2025: 49; 2026: 20.

309 papers.

Sort:

By area:

  1. 2022Visual Prompt Tuningcited by 1,674
  2. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  6. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  7. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  8. 2022Learning Video Representations from Large Language Modelscited by 204
  9. 2023Video Understanding with Large Language Models: A Surveycited by 203
  10. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  11. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  12. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
38 more
  1. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  2. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  3. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  4. 2022Unified Vision and Language Prompt Learningcited by 145
  5. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  6. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  7. 2023VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detectioncited by 121
  8. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  9. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  10. 2023Implicit Neural Representation for Cooperative Low-light Image Enhancementcited by 92
  11. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 90
  12. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  13. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  14. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  15. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
  16. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  17. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 70
  18. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  19. 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
  20. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  21. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 66
  22. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66
  23. 2023Segment Any Anomaly without Training via Hybrid Prompt Regularizationcited by 59
  24. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learningcited by 58
  25. 2023Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferringcited by 57
  26. 2023Open-Vocabulary Video Anomaly Detectioncited by 57
  27. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  28. 2023DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discoverycited by 53
  29. 2022Bayesian Prompt Learning for Image-Language Model Generalizationcited by 51
  30. 2022Multitask Vision-Language Prompt Tuningcited by 51
  31. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  32. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  33. 2023AutoAD: Movie Description in Contextcited by 50
  34. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  35. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 48
  36. 2023Implicit Temporal Modeling with Learnable Alignment for Video Recognitioncited by 44
  37. 2022Generative Action Description Prompts for Skeleton-based Action Recognitioncited by 43
  38. 2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43

50 most cited of 309 All 309 (JSON)