mlsanityData to 7 Oct 2026

Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection

arXiv 2302.00268 ↗First posted 1 Feb 2023cs.CVcited by 22

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Prompt tuning with large-scale pretrained vision-language models empowers open-vocabulary predictions trained on limited base categories, e.g., object classification and detection. In this paper, we propose compositional prompt tuning with motion cues: an extended prompt tuning paradigm for compositional predictions of video data. In particular, we present Relation Prompt (RePro) for Open-vocabulary Video Visual Relation Detection (Open-VidVRD), where conventional prompt tuning is easily biased to certain subject-object combinations and motion patterns. To this end, RePro addresses the two technical challenges of Open-VidVRD: 1) the prompt tokens should respect the two different semantic roles of subject and object, and 2) the tuning should account for the diverse spatio-temporal motion patterns of the subject-object compositions. Without bells and whistles, our RePro achieves a new state-of-the-art performance on two VidVRD benchmarks of not only the base training object and predicate categories, but also the unseen ones. Extensive ablations also demonstrate the effectiveness of the proposed compositional and multi-mode design of prompts. Code is available at https://github.com/Dawn-LX/OpenVoc-VidVRD.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 47 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

36 of 47 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  2. 2021A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models16 Oct
  3. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2017Mask R-CNNcited by 7,259
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2017Simple Online and Realtime Tracking with a Deep Association Metriccited by 908
  8. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  9. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  10. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  11. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  12. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
21 more
  1. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  2. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  3. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  4. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  5. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  6. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  7. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
  8. 2016Seq-NMS for Video Object Detectioncited by 68
  9. 2022Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillationcited by 67
  10. 2022Zero-Shot Temporal Action Detection via Vision-Language Promptingcited by 65
  11. 2022The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generationcited by 60
  12. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  13. 2018Counterfactual Critic Multi-Agent Training for Scene Graph Generationcited by 53
  14. 2021Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphscited by 23
  15. 2021Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integrationcited by 21
  16. 2022Robust Region Feature Synthesizer for Zero-Shot Object Detectioncited by 20
  17. 2022Integrating Object-aware and Interaction-aware Knowledge for Weakly Supervised Scene Graph Generationcited by 20
  18. 2022NICEST: Noisy Label Correction and Training for Robust Scene Graph Generationcited by 19
  19. 2021Social Fabric: Tubelet Compositions for Video Relation Detectioncited by 18
  20. 2021Video Relation Detection via Tracklet based Visual Transformercited by 17
  21. 2022Rethinking the Evaluation of Unbiased Scene Graph Generationcited by 9

Built on by

Cited by year

Citing papers per year: 2023: 9; 2024: 7; 2025: 6.

22 papers.

Sort:

By area:

  1. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  2. 2023Visual Tuningcited by 34
  3. 2023Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Modelscited by 27
  4. 2023UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memorycited by 26
  5. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  6. 2025DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformerscited by 17
  7. 2024Scene Graph Generation with Role-Playing Large Language Modelscited by 16
  8. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  9. 2025Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptationcited by 10
  10. 2024Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learningcited by 6
  11. 2024Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Roomscited by 6
  12. 2023Decomposed Prototype Learning for Few-Shot Scene Graph Generationcited by 4
10 more
  1. 2024Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuningcited by 4
  2. 2025Weakly Supervised Video Scene Graph Generation via Natural Language Supervisioncited by 4
  3. 2023Taking A Closer Look at Visual Relation: Unbiased Video Scene Graph Generation with Decoupled Label Learningcited by 3
  4. 2024End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Promptingcited by 3
  5. 2025TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferringcited by 3
  6. 2025MoLT: Mixture of Layer-Wise Tokens for Efficient Audio-Visual Learningcited by 2
  7. 2023Open-Vocabulary Video Relation Extractioncited by 1
  8. 2024TD^2-Net: Toward Denoising and Debiasing for Dynamic Scene Graph Generationcited by 1
  9. 2024Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancementcited by 1
  10. 2025METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detectioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenVidVRD: Open-Vocabulary Video Visual Relation Detection via Prompt-Driven Semantic Space Alignment
  2. 2025ART: Adaptive Relation Tuning for Generalized Relation Prediction
  3. 2023Knowledge-Aware Prompt Tuning for Generalizable Vision-Language Models
  4. 2025GA2-CLIP: Generic Attribute Anchor for Efficient Prompt Tuningin Video-Language Models
  5. 2025SDVPT: Semantic-Driven Visual Prompt Tuning for Open-World Object Counting
  6. 2024Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation
  7. 2025OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
  8. 2024Generalizable Prompt Tuning for Vision-Language Models