mlsanityData to 7 Oct 2026

MDMMT: Multidomain Multimodal Transformer for Video Retrieval

arXiv 2103.10699 ↗First posted 19 Mar 2021cs.CVcited by 78

Not on the map. Why?

Abstract (arXiv)

We present a new state-of-the-art on the text to video retrieval task on MSRVTT and LSMDC benchmarks where our model outperforms all previous solutions by a large margin. Moreover, state-of-the-art results are achieved with a single model on two datasets without finetuning. This multidomain generalisation is achieved by a proper combination of different video caption datasets. We show that training on different datasets can improve test results of each other. Additionally we check intersection between many popular datasets and found that MSRVTT has a significant overlap between the test and the train parts, and the same situation is observed for ActivityNet.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 19 Mar 2021.

Builds on

35 of 50 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2017The Kinetics Human Action Video Datasetcited by 1,894
  5. 2018SlowFast Networks for Video Recognitioncited by 1,862
  6. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  7. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  8. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  9. 2017Dense-Captioning Events in Videoscited by 789
  10. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  11. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  12. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
23 more
  1. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  2. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  3. 2020Multi-modal Transformer for Video Retrievalcited by 395
  4. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  5. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  6. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  7. 2015A Dataset for Movie Descriptioncited by 195
  8. 2016Movie Descriptioncited by 180
  9. 2020Support-set bottlenecks for video-text representation learningcited by 170
  10. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
  11. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  12. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  13. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  14. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  15. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  16. 2020A Short Note on the Kinetics-700-2020 Human Action Datasetcited by 89
  17. 2018Dual Encoding for Zero-Example Video Retrievalcited by 85
  18. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  19. 2016End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answeringcited by 60
  20. 2017Predicting Visual Features from Text for Image and Video Caption Retrievalcited by 53
  21. 2016Learning Language-Visual Embedding for Movie Understanding with Natural-Languagecited by 41
  22. 2018Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object Interactioncited by 40
  23. 2015Using Descriptive Video Services to Create a Large Data Source for Video Annotation Researchcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 27; 2023: 17; 2024: 13; 2025: 4; 2026: 2.

78 papers.

Sort:

By area:

  1. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  2. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  3. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  4. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  5. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  6. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  7. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  8. 2021CycleMLP: A MLP-like Architecture for Dense Predictioncited by 126
  9. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  10. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  11. 2021Cross-Modality Fusion Transformer for Multispectral Object Detectioncited by 86
  12. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
38 more
  1. 2022Video Transformers: A Surveycited by 63
  2. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  3. 2022Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videoscited by 50
  4. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  5. 2022ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cuescited by 42
  6. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
  7. 2021Cross-Modal Discrete Representation Learningcited by 35
  8. 2023Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Modelscited by 34
  9. 2024Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrievalcited by 33
  10. 2021Masking Modalities for Cross-modal Video Retrievalcited by 23
  11. 2023Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrievalcited by 22
  12. 2022Integrating Language Guidance into Vision-based Deep Metric Learningcited by 19
  13. 2022Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representationscited by 19
  14. 2023Tagging before Alignment: Integrating Multi-Modal Tags for Video-Text Retrievalcited by 18
  15. 2022Learning to Retrieve Videos by Asking Questionscited by 17
  16. 2022MIMO Is All You Need : A Strong Multi-In-Multi-Out Baseline for Video Predictioncited by 17
  17. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 17
  18. 2022Learn to Understand Negation in Video Retrievalcited by 16
  19. 2023Deep Learning for Video-Text Retrieval: a Reviewcited by 15
  20. 2021CLIP2TV: Align, Match and Distill for Video-Text Retrievalcited by 14
  21. 2022SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-trainingcited by 14
  22. 2023Reinforcement Learning Friendly Vision-Language Model for Minecraftcited by 14
  23. 2021Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrievalcited by 13
  24. 2023Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detectioncited by 13
  25. 2021Connecting Language and Vision for Natural Language-Based Vehicle Retrievalcited by 12
  26. 2022Multi-Query Video Retrievalcited by 11
  27. 2022MDMMT-2: Multidomain Multimodal Transformer for Video Retrieval, One More Step Towards Generalizationcited by 10
  28. 2022Learning to Locate Visual Answer in Video Corpus Using Questioncited by 10
  29. 2022Boosting Video-Text Retrieval with Explicit High-Level Semanticscited by 8
  30. 2023Video-Text Retrieval by Supervised Sparse Multi-Grained Learningcited by 7
  31. 2022Learning video retrieval models with relevance-aware online miningcited by 6
  32. 2022Relevance-based Margin for Contrastively-trained Video Retrieval Modelscited by 6
  33. 2023Weakly-Supervised HOI Detection from Interaction Labels Only and Language/Vision-Language Priorscited by 6
  34. 2021AudioVisual Video Summarizationcited by 5
  35. 2022RaP: Redundancy-aware Video-language Pre-training for Text-Video Retrievalcited by 5
  36. 2023STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-trainingcited by 5
  37. 2023Mask to reconstruct: Cooperative Semantics Completion for Video-text Retrievalcited by 5
  38. 2023In-Style: Bridging Text and Uncurated Videos with Style Transfer for Text-Video Retrievalcited by 5

50 most cited of 78 All 78 (JSON)