mlsanityData to 7 Oct 2026

ConTra: (Con)text (Tra)nsformer for Cross-Modal Video Retrieval

arXiv 2210.04341 ↗First posted 9 Oct 2022cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

In this paper, we re-examine the task of cross-modal clip-sentence retrieval, where the clip is part of a longer untrimmed video. When the clip is short or visually ambiguous, knowledge of its local temporal context (i.e. surrounding video segments) can be used to improve the retrieval performance. We propose Context Transformer (ConTra); an encoder architecture that models the interaction between a video clip and its local temporal context in order to enhance its embedded representations. Importantly, we supervise the context transformer using contrastive losses in the cross-modal embedding space. We explore context transformers for video and text modalities. Results consistently demonstrate improved performance on three datasets: YouCook2, EPIC-KITCHENS and a clip-sentence version of ActivityNet Captions. Exhaustive ablation studies and context analysis show the efficacy of the proposed method.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 55 of its 73 references that are in the map. First posted on arXiv 9 Oct 2022.

Builds on

55 of 73 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  5. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  10. 2017Dense-Captioning Events in Videoscited by 789
  11. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  12. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
38 more
  1. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  2. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  3. 2020Rescaling Egocentric Visioncited by 529
  4. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  5. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  6. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  7. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  8. 2020Multi-modal Transformer for Video Retrievalcited by 395
  9. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  10. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  11. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  12. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  13. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  14. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  15. 2014Learning Fine-grained Image Similarity with Deep Rankingcited by 189
  16. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  17. 2016Exploring the Limits of Language Modelingcited by 172
  18. 2020Support-set bottlenecks for video-text representation learningcited by 170
  19. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 150
  20. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  21. 2019Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagationcited by 125
  22. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  23. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  24. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  25. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  26. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  27. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  28. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  29. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  30. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  31. 2021Training Vision Transformers for Image Retrievalcited by 90
  32. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  33. 2018Localizing Moments in Video with Temporal Languagecited by 83
  34. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  35. 2021Temporal Query Networks for Fine-grained Video Understandingcited by 68
  36. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
  37. 2019Context R-CNN: Long Term Temporal Context for Per-Camera Object Detectioncited by 45
  38. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Video Editing for Video Retrievalcited by 1
  2. 2025Leveraging Modality Tags for Enhanced Cross-Modal Video Retrievalcited by 1
  3. 2025Leveraging Auxiliary Information in Text-to-Video Retrieval: A Reviewcited by 1