mlsanityData to 7 Oct 2026

Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval

arXiv 2503.19009 ↗First posted 24 Mar 2025cs.CV cs.IRcited by 20

Not on the map. Why?

Abstract (arXiv)

In this work, we tackle the problem of text-to-video retrieval (T2VR). Inspired by the success of late interaction techniques in text-document, text-image, and text-video retrieval, our approach, Video-ColBERT, introduces a simple and efficient mechanism for fine-grained similarity assessment between queries and videos. Video-ColBERT is built upon 3 main components: a fine-grained spatial and temporal token-wise interaction, query and visual expansions, and a dual sigmoid loss during training. We find that this interaction and training paradigm leads to strong individual, yet compatible, representations for encoding video content. These representations lead to increases in performance on common text-to-video retrieval benchmarks compared to other bi-encoder methods.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 54 of its 67 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

54 of 67 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
38 more
  1. 2017Localizing Moments in Video with Natural Languagecited by 660
  2. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  3. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  4. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  5. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  6. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  7. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  8. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  9. 2020Multi-modal Transformer for Video Retrievalcited by 395
  10. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  11. 2020Debiased Contrastive Learningcited by 354
  12. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  13. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  14. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  15. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  16. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  17. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  18. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  19. 2020Support-set bottlenecks for video-text representation learningcited by 170
  20. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  21. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  22. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  23. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  24. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  25. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  26. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  27. 2022Disentangled Representation Learning for Text-Video Retrievalcited by 77
  28. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  29. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  30. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  31. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  32. 2022Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representationscited by 57
  33. 2023UATVR: Uncertainty-Adaptive Text-Video Retrievalcited by 55
  34. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  35. 2021On Semantic Similarity in Video Retrievalcited by 45
  36. 2023Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrievalcited by 35
  37. 2022Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representationscited by 19
  38. 2021Video-aided Unsupervised Grammar Inductioncited by 17

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 14.

20 papers.

Sort:

By area:

  1. 2025WikiVideo: Article Generation from Multiple Videoscited by 13
  2. 2026Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrievalcited by 3
  3. 2026SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memorycited by 3
  4. 2025Simple Projection Variants Improve ColBERT Performancecited by 2
  5. 2026Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrievalcited by 2
  6. 2025CMRAG: Co-modality-based visual document retrieval and question answeringcited by 1
  7. 2025Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generationcited by 1
  8. 2025LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026cited by 1
  9. 2026Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coveragecited by 1
  10. 2026Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understandingcited by 1
  11. 2026Binding Multiple Modalities via Multimodal Wasserstein Barycentercited by 1
  12. 2025T2VParser: Adaptive Decomposition Tokens for Partial Alignment in Text to Video Retrievalcited by 0
8 more
  1. 2026SAVE: Speech-Aware Video Representation Learning for Video-Text Retrievalcited by 0
  2. 2026AMES: Approximate Multi-modal Enterprise Search via Late Interaction Retrievalcited by 0
  3. 2026GEM: A Native Graph-based Index for Multi-Vector Retrievalcited by 0
  4. 2026EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrievalcited by 0
  5. 2026MARQUIS: A Three-Stage Pipeline for Video Retrieval-Augmented Generationcited by 0
  6. 2026Distribution-Alignment Bridge for Uncertainty-Aware Text-to-Video Retrievalcited by 0
  7. 2026MultiVENT-Raw: A Benchmark for Retrieval and Reasoning over Raw Videoscited by 0
  8. 2026SALI: Shot-Aware Late Interaction for Cross-Shot Relation Matching in Text-to-Video Retrieval using Film-Grammar Knowledgecited by 0