# Memory Enhanced Embedding Learning for Cross-Modal Video-Text Retrieval

arXiv: https://arxiv.org/abs/2103.15686 · HTML: https://mlsanity.com/paper/2103.15686/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 29 Mar 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [Use What You Have: Video Retrieval Using Representations From Collaborative Experts](https://mlsanity.com/paper/1907.13487/) (2019)
- [A Joint Sequence Fusion Model for Video Question Answering and Retrieval](https://mlsanity.com/paper/1808.02559/) (2018)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [Visual Semantic Reasoning for Image-Text Matching](https://mlsanity.com/paper/1909.02701/) (2019)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [Learning to Assemble Neural Module Tree Networks for Visual Grounding](https://mlsanity.com/paper/1812.03299/) (2018)
- [Learning Deep Structure-Preserving Image-Text Embeddings](https://mlsanity.com/paper/1511.06078/) (2015)
- [Invariance Matters: Exemplar Memory for Domain Adaptive Person Re-identification](https://mlsanity.com/paper/1904.01990/) (2019)
- [CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval](https://mlsanity.com/paper/1909.05506/) (2019)
- [Dual Encoding for Zero-Example Video Retrieval](https://mlsanity.com/paper/1809.06181/) (2018)
- [Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval](https://mlsanity.com/paper/1906.04402/) (2019)
- [Learning Semantic Concepts and Order for Image and Sentence Matching](https://mlsanity.com/paper/1712.02036/) (2017)
- [Identity-Aware Textual-Visual Matching with Latent Co-attention](https://mlsanity.com/paper/1708.01988/) (2017)
- [Predicting Visual Features from Text for Image and Video Caption Retrieval](https://mlsanity.com/paper/1709.01362/) (2017)
- [Instance-aware Image and Sentence Matching with Selective Multimodal LSTM](https://mlsanity.com/paper/1611.05588/) (2016)
- [Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding](https://mlsanity.com/paper/1908.10568/) (2019)
- [Context-Aware Visual Policy Network for Fine-Grained Image Captioning](https://mlsanity.com/paper/1906.02365/) (2019)
- [Context-Aware Visual Policy Network for Sequence-Level Image Captioning](https://mlsanity.com/paper/1808.05864/) (2018)

## Built on by (most cited first, 1 of 1)

- [Boosting Video-Text Retrieval with Explicit High-Level Semantics](https://mlsanity.com/paper/2208.04215/) (2022)

## Abstract (arXiv)

Cross-modal video-text retrieval, a challenging task in the field of vision and language, aims at retrieving corresponding instance giving sample from either modality. Existing approaches for this task all focus on how to design encoding model through a hard negative ranking loss, leaving two key problems unaddressed during this procedure. First, in the training stage, only a mini-batch of instance pairs is available in each iteration. Therefore, this kind of hard negatives is locally mined inside a mini-batch while ignoring the global negative samples among the dataset. Second, there are many text descriptions for one video and each text only describes certain local features of a video. Previous works for this task did not consider to fuse the multiply texts corresponding to a video during the training. In this paper, to solve the above two problems, we propose a novel memory enhanced embedding learning (MEEL) method for videotext retrieval. To be specific, we construct two kinds of memory banks respectively: cross-modal memory module and text center memory module. The cross-modal memory module is employed to record the instance embeddings of all the datasets for global negative mining. To avoid the fast evolving of the embedding in the memory bank during training, we utilize a momentum encoder to update the features by a moving-averaging strategy. The text center memory module is designed to record the center information of the multiple textual instances corresponding to a video, and aims at bridging these textual instances together. Extensive experimental results on two challenging benchmarks, i.e., MSR-VTT and VATEX, demonstrate the effectiveness of the proposed method.