mlsanityData to 7 Oct 2026

VicTR: Video-conditioned Text Representations for Activity Recognition

arXiv 2304.02560 ↗First posted 5 Apr 2023cs.CVcited by 28

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Vision-Language models (VLMs) have excelled in the image-domain -- especially in zero-shot settings -- thanks to the availability of vast pretraining data (i.e., paired image-text samples). However for videos, such paired data is not as abundant. Therefore, video-VLMs are usually designed by adapting pretrained image-VLMs to the video-domain, instead of training from scratch. All such recipes rely on augmenting visual embeddings with temporal information (i.e., image rightarrow video), often keeping text embeddings unchanged or even being discarded. In this paper, we argue the contrary, that better video-VLMs can be designed by focusing more on augmenting text, rather than visual information. More specifically, we introduce Video-conditioned Text Representations (VicTR): a form of text embeddings optimized w.r.t. visual embeddings, creating a more-flexible contrastive latent space. Our model can further make use of freely-available semantic information, in the form of visually-grounded auxiliary text (e.g. object or scene information). We evaluate our model on few-shot, zero-shot (HMDB-51, UCF-101), short-form (Kinetics-400) and long-form (Charades) activity recognition benchmarks, showing strong performance among video-VLMs.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 84 of its 101 references that are in the map. First posted on arXiv 5 Apr 2023.

Builds on

84 of 101 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning27 Feb

Other references · 83

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2017Non-local Neural Networkscited by 2,290
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2018SlowFast Networks for Video Recognitioncited by 1,862
  11. 2021ViViT: A Video Vision Transformercited by 1,610
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
38 more
  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  4. 2021Video Swin Transformercited by 1,090
  5. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  6. 2021Multiscale Vision Transformerscited by 896
  7. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  10. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  11. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  12. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  13. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  14. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  15. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  16. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  17. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  18. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  19. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  20. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  21. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  22. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  23. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  24. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  25. 2022Visual Classification via Description from Large Language Modelscited by 331
  26. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  27. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  28. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  29. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  30. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  31. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  32. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  33. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  34. 2022Egocentric Video-Language Pretrainingcited by 231
  35. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  36. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  37. 2022Learning Video Representations from Large Language Modelscited by 204
  38. 2023Teaching CLIP to Count to Tencited by 194

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 13; 2026: 2.

28 papers.

Sort:

By area:

  1. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  2. 2024Language Repository for Long Video Understandingcited by 55
  3. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  4. 2024XOV-Action: Towards Generalizable Open-Vocabulary Action Recognitioncited by 15
  5. 2024HandsOnVLM: Vision-Language Models for Hand-Object Interaction Predictioncited by 15
  6. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  7. 2024A Comprehensive Review of Few-shot Action Recognitioncited by 11
  8. 2024Event Stream based Human Action Recognition: A High-Definition Benchmark Dataset and Algorithmscited by 11
  9. 2025Object-Shot Enhanced Grounding Network for Egocentric Videocited by 11
  10. 2023Language-based Action Concept Spaces Improve Video Self-Supervised Learningcited by 10
  11. 2024Building a Multi-modal Spatiotemporal Expert for Zero-shot Action Recognition with CLIPcited by 9
  12. 2023Generating Action-conditioned Prompts for Open-vocabulary Video Action Recognitioncited by 7
16 more
  1. 2025Learning to Generalize without Bias for Open-Vocabulary Action Recognitioncited by 7
  2. 2024Continual Learning Improves Zero-Shot Action Recognitioncited by 5
  3. 2024Learning Causal Domain-Invariant Temporal Dynamics for Few-Shot Action Recognitioncited by 4
  4. 2024A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learningcited by 2
  5. 2025TEST-V: TEst-time Support-set Tuning for Zero-shot Video Classificationcited by 2
  6. 2025VTD-CLIP: Video-to-Text Discretization via Prompting CLIPcited by 2
  7. 2023AAN: Attributes-Aware Network for Temporal Action Detectioncited by 1
  8. 2025IoT-Based Real-Time Medical-Related Human Activity Recognition Using Skeletons and Multi-Stage Deep Learning for Healthcarecited by 1
  9. 2025Leveraging Modality Tags for Enhanced Cross-Modal Video Retrievalcited by 1
  10. 2025SPKLIP: Aligning Spike Video Streams with Natural Languagecited by 1
  11. 2025Prompt-guided Disentangled Representation for Action Recognitioncited by 1
  12. 2025Enhancing Spatio-Temporal Zero-shot Action Recognition with Language-driven Description Attributescited by 1
  13. 2025StretchySnake: Flexible SSM Training Unlocks Action Recognition Across Spatio-Temporal Scalescited by 0
  14. 2025Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challengescited by 0
  15. 2026Knowledge-guided Disentanglement with Atomic Actions for Action Recognitioncited by 0
  16. 2026TF-CADE: Foreground-Concentrated Text-Video Alignment for Zero-Shot Temporal Action Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Analyzing Zero-Shot Abilities of Vision-Language Models on Video Understanding Tasks
  2. 2023Videoprompter: an ensemble of foundational models for zero-shot video understanding
  3. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  4. 2026VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval
  5. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
  6. 2023Vamos: Versatile Action Models for Video Understanding
  7. 2024Enhancing Video Transformers for Action Understanding with VLM-aided Training
  8. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models