mlsanityData to 7 Oct 2026

X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval

arXiv 2207.07285 ↗First posted 15 Jul 2022cs.CVcited by 279

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video-text retrieval has been a crucial and fundamental task in multi-modal research. The development of video-text retrieval has been considerably promoted by large-scale multi-modal contrastive pre-training, which primarily focuses on coarse-grained or fine-grained contrast. However, cross-grained contrast, which is the contrast between coarse-grained representations and fine-grained representations, has rarely been explored in prior research. Compared with fine-grained or coarse-grained contrasts, cross-grained contrast calculate the correlation between coarse-grained features and each fine-grained feature, and is able to filter out the unnecessary fine-grained features guided by the coarse-grained feature during similarity calculation, thus improving the accuracy of retrieval. To this end, this paper presents a novel multi-grained contrastive model, namely X-CLIP, for video-text retrieval. However, another challenge lies in the similarity aggregation problem, which aims to aggregate fine-grained and cross-grained similarity matrices to instance-level similarity. To address this challenge, we propose the Attention Over Similarity Matrix (AOSM) module to make the model focus on the contrast between essential frames and words, thus lowering the impact of unnecessary frames and words on retrieval results. With multi-grained contrast and the proposed AOSM module, X-CLIP achieves outstanding performance on five widely-used video-text retrieval datasets, including MSR-VTT (49.3 R@1), MSVD (50.4 R@1), LSMDC (26.1 R@1), DiDeMo (47.8 R@1) and ActivityNet (46.2 R@1). It outperforms the previous state-of-theart by +6.3%, +6.6%, +11.1%, +6.7%, +3.8% relative improvements on these benchmarks, demonstrating the superiority of multi-grained contrast and AOSM.

Key facts. Cited by 279 papers in the map (data through 7 Oct 2026). Builds on 56 of its 67 references that are in the map. First posted on arXiv 15 Jul 2022.

Builds on

56 of 67 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
38 more
  1. 2021ViViT: A Video Vision Transformercited by 1,610
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  7. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  8. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  9. 2017Localizing Moments in Video with Natural Languagecited by 660
  10. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  11. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  12. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  13. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  14. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  15. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  16. 2020Multi-modal Transformer for Video Retrievalcited by 395
  17. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  18. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  19. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  20. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  21. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  22. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  23. 2020Support-set bottlenecks for video-text representation learningcited by 170
  24. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  25. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  26. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  27. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  28. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  29. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  30. 2014Translating Videos to Natural Language Using Deep Recurrent Neural Networkscited by 95
  31. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  32. 2018Dual Encoding for Zero-Example Video Retrievalcited by 85
  33. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  34. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 84
  35. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  36. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  37. 2020Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learningcited by 74
  38. 2020Relevance-guided Supervision for OpenQA with ColBERTcited by 71

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 62; 2024: 76; 2025: 77; 2026: 52.

279 papers.

Sort:

By area:

  1. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  2. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  3. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  4. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  5. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  6. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  7. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  8. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  9. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  10. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  11. 2022CLIP-Driven Fine-grained Text-Image Person Re-identificationcited by 79
  12. 2023Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentationcited by 76
38 more
  1. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  2. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
  3. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  4. 2023DePT: Decoupled Prompt Tuningcited by 58
  5. 2023UATVR: Uncertainty-Adaptive Text-Video Retrievalcited by 55
  6. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  7. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 55
  8. 2023DiffusionRet: Generative Text-Video Retrieval with Diffusion Modelcited by 52
  9. 2023CoVR-2: Automatic Data Construction for Composed Video Retrievalcited by 52
  10. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  11. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  12. 2024Cross-Modality Perturbation Synergy Attack for Person Re-identificationcited by 45
  13. 2022Clover: Towards A Unified Video-Language Alignment and Fusion Modelcited by 41
  14. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  15. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 40
  16. 2023X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidancecited by 37
  17. 2022VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrievalcited by 36
  18. 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36
  19. 2022Cross-Modal Adapter for Vision-Language Retrievalcited by 35
  20. 2023Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrievalcited by 35
  21. 2022LidarCLIP or: How I Learned to Talk to Point Cloudscited by 33
  22. 2023EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognitioncited by 31
  23. 2023iKUN: Speak to Trackers without Retrainingcited by 31
  24. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 31
  25. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29
  26. 2023VicTR: Video-conditioned Text Representations for Activity Recognitioncited by 28
  27. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  28. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  29. 2025Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generationcited by 27
  30. 2023Towards Local Visual Modeling for Image Captioningcited by 26
  31. 2023ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Modelscited by 26
  32. 2024DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrievalcited by 26
  33. 2024Composed Video Retrieval via Enriched Context and Discriminative Embeddingscited by 26
  34. 2023Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Modelcited by 25
  35. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  36. 2026Delving Deeper: Hierarchical Visual Perception for Robust Video-Text Retrievalcited by 23
  37. 2024GaussianVTON: 3D Human Virtual Try-ON via Multi-Stage Gaussian Splatting Editing with Image Promptingcited by 22
  38. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22

50 most cited of 279 All 279 (JSON)