mlsanityData to 7 Oct 2026

Verbs in Action: Improving verb understanding in video-language models

arXiv 2304.06708 ↗First posted 13 Apr 2023cs.CV cs.AI cs.CLcited by 76

Not on the map. Why?

Abstract (arXiv)

Understanding verbs is crucial to modelling how people and objects interact with each other and the environment through space and time. Recently, state-of-the-art video-language models based on CLIP have been shown to have limited verb understanding and to rely extensively on nouns, restricting their performance in real-world video applications that require action and temporal understanding. In this work, we improve verb understanding for CLIP-based video-language models by proposing a new Verb-Focused Contrastive (VFC) framework. This consists of two main components: (1) leveraging pretrained large language models (LLMs) to create hard negatives for cross-modal contrastive learning, together with a calibration strategy to balance the occurrence of concepts in positive and negative pairs; and (2) enforcing a fine-grained, verb phrase alignment loss. Our method achieves state-of-the-art results for zero-shot performance on three downstream tasks that focus on verb understanding: video-text matching, video question-answering and video classification. To the best of our knowledge, this is the first work which proposes a method to alleviate the verb understanding problem, and does not simply highlight it.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 84 of its 111 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

84 of 111 references.

References · 84

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
38 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  2. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  3. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  5. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  6. 2020Contrastive Learning with Hard Negative Samplescited by 619
  7. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  8. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  11. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  12. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  13. 2020Multi-modal Transformer for Video Retrievalcited by 395
  14. 2017Temporal Relational Reasoning in Videoscited by 324
  15. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  16. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  17. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  18. 2017Sampling Matters in Deep Embedding Learningcited by 285
  19. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  20. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  21. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  22. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  23. 2022Learning Video Representations from Large Language Modelscited by 204
  24. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  25. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  26. 2016Movie Descriptioncited by 180
  27. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  28. 2020SpeedNet: Learning the Speediness in Videoscited by 171
  29. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  30. 2018Self-Supervised Video Representation Learning with Space-Time Cubic Puzzlescited by 149
  31. 2022Multiview Transformers for Video Recognitioncited by 148
  32. 2020Self-supervised Video Representation Learning by Pace Predictioncited by 139
  33. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  34. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  35. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
  36. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  37. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  38. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 33; 2025: 17; 2026: 8.

76 papers.

Sort:

By area:

  1. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  2. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  3. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  4. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  5. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  6. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 72
  7. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  8. 2024Language Repository for Long Video Understandingcited by 55
  9. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
  10. 2024FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognitioncited by 35
  11. 2023Collaborating Foundation Models for Domain Generalized Semantic Segmentationcited by 33
  12. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29
38 more
  1. 2023The devil is in the fine-grained details: Evaluating open-vocabulary object detectors for fine-grained understandingcited by 29
  2. 2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25
  3. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  4. 2023SPOT! Revisiting Video-Language Models for Event Understandingcited by 20
  5. 2023Lego: Learning to Disentangle and Invert Personalized Concepts Beyond Object Appearance in Text-to-Image Diffusion Modelscited by 19
  6. 2024Test-Time Zero-Shot Temporal Action Localizationcited by 19
  7. 2024Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognitioncited by 17
  8. 2024Semantically Controllable Augmentations for Generalizable Robot Learningcited by 17
  9. 2023OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognitioncited by 16
  10. 2024XOV-Action: Towards Generalizable Open-Vocabulary Action Recognitioncited by 15
  11. 2025VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Modelscited by 12
  12. 2024Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?cited by 11
  13. 2024Question-Answering Dense Video Eventscited by 11
  14. 2024Progress-Aware Video Frame Captioningcited by 10
  15. 2025Chirality in Action: Time-Aware Video Representation Learning by Latent Straighteningcited by 10
  16. 2023Long Video Understanding with Learnable Retrieval in Video-Language Modelscited by 9
  17. 2023Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusioncited by 8
  18. 2024Question-Instructed Visual Descriptions for Zero-Shot Video Question Answeringcited by 8
  19. 2024Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detectioncited by 8
  20. 2024Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic Viewcited by 7
  21. 2025VerbDiff: Text-Only Diffusion Models with Enhanced Interaction Awarenesscited by 7
  22. 2025RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graphcited by 7
  23. 2024Using Left and Right Brains Together: Towards Vision and Language Planningcited by 6
  24. 2024Text-Enhanced Zero-Shot Action Recognition: A training-free approachcited by 6
  25. 2024Towards Long Video Understanding via Fine-detailed Video Story Generationcited by 6
  26. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  27. 2025OSLoPrompt: Bridging Low-Supervision Challenges and Open-Set Domain Generalization in CLIPcited by 6
  28. 2023LEAP: LLM-Generation of Egocentric Action Programscited by 5
  29. 2023Vision-Language Models Learn Super Images for Efficient Partially Relevant Video Retrievalcited by 5
  30. 2024VidLA: Video-Language Alignment at Scalecited by 5
  31. 2024Open-Vocabulary Spatio-Temporal Action Detectioncited by 5
  32. 2024Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answeringcited by 5
  33. 2024Web-Scale Visual Entity Recognition: An LLM-Driven Data Approachcited by 5
  34. 2026Towards Unified Vision-Language Models with Incomplete Multi-Modal Inputscited by 5
  35. 2023Defining a New NLP Playgroundcited by 4
  36. 2024SRL-CLIP: Efficient CLIP Video Adaptation via Structured Semantic Role Labelscited by 4
  37. 2025Expertized Caption Auto-Enhancement for Video-Text Retrievalcited by 4
  38. 2025FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Modelscited by 4

50 most cited of 76 All 76 (JSON)