mlsanityData to 7 Oct 2026

Multi-modal Transformer for Video Retrieval

arXiv 2007.10639 ↗First posted 21 Jul 2020cs.CVcited by 395

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The task of retrieving video content relevant to natural language queries plays a critical role in effectively handling internet-scale datasets. Most of the existing methods for this caption-to-video retrieval problem do not fully exploit cross-modal cues present in video. Furthermore, they aggregate per-frame visual features with limited or no temporal information. In this paper, we present a multi-modal transformer to jointly encode the different modalities in video, which allows each of them to attend to the others. The transformer architecture is also leveraged to encode and model the temporal information. On the natural language side, we investigate the best practices to jointly optimize the language embedding together with the multi-modal transformer. This novel framework allows us to establish state-of-the-art results for video retrieval on three datasets. More details are available at http://thoth.inrialpes.fr/research/MMT.

Key facts. Cited by 395 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 82; 2022: 111; 2023: 85; 2024: 62; 2025: 36; 2026: 19.

395 papers.

Sort:

By area:

  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  2. 2021Multiscale Vision Transformerscited by 896
  3. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  4. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  5. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  6. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  7. 2022Multimodal Learning with Transformers: A Surveycited by 399
  8. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  9. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  10. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  11. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  12. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
38 more
  1. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  2. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  3. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  4. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  5. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  6. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  7. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  8. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  9. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  10. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  11. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  12. 2021CycleMLP: A MLP-like Architecture for Dense Predictioncited by 126
  13. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  14. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  15. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  16. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115
  17. 2021VidTr: Video Transformer Without Convolutionscited by 112
  18. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  19. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  20. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  21. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  22. 2021CLIP-It! Language-Guided Video Summarizationcited by 99
  23. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  24. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  25. 2021Interventional Video Grounding with Dual Contrastive Learningcited by 94
  26. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  27. 2021Negative Sample Matters: A Renaissance of Metric Learning for Temporal Groundingcited by 90
  28. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  29. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  30. 2021Cross-Modality Fusion Transformer for Multispectral Object Detectioncited by 86
  31. 2022CenterCLIP: Token Clustering for Efficient Text-Video Retrievalcited by 86
  32. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  33. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  34. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  35. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  36. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  37. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  38. 2022Disentangled Representation Learning for Text-Video Retrievalcited by 77

50 most cited of 395 All 395 (JSON)