mlsanityData to 7 Oct 2026

MDMMT-2: Multidomain Multimodal Transformer for Video Retrieval, One More Step Towards Generalization

arXiv 2203.07086 ↗First posted 14 Mar 2022cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

In this work we present a new State-of-The-Art on the text-to-video retrieval task on MSR-VTT, LSMDC, MSVD, YouCook2 and TGIF obtained by a single model. Three different data sources are combined: weakly-supervised videos, crowd-labeled text-image pairs and text-video pairs. A careful analysis of available pre-trained networks helps to choose the best prior-knowledge ones. We introduce three-stage training procedure that provides high transfer knowledge efficiency and allows to use noisy datasets during training without prior knowledge degradation. Additionally, double positional encoding is used for better fusion of different modalities and a simple method for non-square inputs processing is suggested.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 14 Mar 2022.

Builds on

39 of 51 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2017The Kinetics Human Action Video Datasetcited by 1,894
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  7. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  8. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  9. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  10. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  11. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  12. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
27 more
  1. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  2. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  3. 2020Multi-modal Transformer for Video Retrievalcited by 395
  4. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  5. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  6. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  7. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  8. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  9. 2016Movie Descriptioncited by 180
  10. 2020Support-set bottlenecks for video-text representation learningcited by 170
  11. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
  12. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  13. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  14. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 110
  15. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  16. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  17. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  18. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  19. 2018Dual Encoding for Zero-Example Video Retrievalcited by 85
  20. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  21. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  22. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  23. 2016End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answeringcited by 60
  24. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  25. 2017Predicting Visual Features from Text for Image and Video Caption Retrievalcited by 53
  26. 2020SEA: Sentence Encoder Assembly for Video Retrieval by Textual Queriescited by 26
  27. 2021TRECVID 2020: A comprehensive campaign for evaluating video retrieval tasks across multiple application domainscited by 16

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 3; 2024: 1; 2025: 2.

10 papers.

  1. 2022Clover: Towards A Unified Video-Language Alignment and Fusion Modelcited by 41
  2. 2022Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representationscited by 19
  3. 2023Meta-Personalizing Vision-Language Models to Find Named Instances in Videocited by 16
  4. 2023Deep Learning for Video-Text Retrieval: a Reviewcited by 15
  5. 2022Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attentioncited by 7
  6. 2023STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-trainingcited by 5
  7. 2022M2HF: Multi-level Multi-modal Hybrid Fusion for Text-Video Retrievalcited by 1
  8. 2025Leveraging Modality Tags for Enhanced Cross-Modal Video Retrievalcited by 1
  9. 2025Leveraging Auxiliary Information in Text-to-Video Retrieval: A Reviewcited by 1
  10. 2024Towards Holistic Language-video Representation: the language model-enhanced MSR-Video to Text Datasetcited by 0