# Frame-wise Cross-modal Matching for Video Moment Retrieval

arXiv: https://arxiv.org/abs/2009.10434 · HTML: https://mlsanity.com/paper/2009.10434/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2020.

## Built on by (most cited first, 26 of 26)

- [Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video](https://mlsanity.com/paper/2203.06667/) (2022)
- [Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval](https://mlsanity.com/paper/2209.11572/) (2022)
- [You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos](https://mlsanity.com/paper/2303.07863/) (2023)
- [Hierarchical Local-Global Transformer for Temporal Sentence Grounding](https://mlsanity.com/paper/2208.14882/) (2022)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding](https://mlsanity.com/paper/2203.02966/) (2022)
- [A Survey on Video Moment Localization](https://mlsanity.com/paper/2306.07515/) (2023)
- [Entity-aware and Motion-aware Transformers for Language-driven Action Localization in Videos](https://mlsanity.com/paper/2205.05854/) (2022)
- [Learning to Locate Visual Answer in Video Corpus Using Question](https://mlsanity.com/paper/2210.05423/) (2022)
- [Context-Enhanced Video Moment Retrieval with Large Language Models](https://mlsanity.com/paper/2405.12540/) (2024)
- [Self-supervised Learning for Semi-supervised Temporal Language Grounding](https://mlsanity.com/paper/2109.11475/) (2021)
- [Visual Answer Localization with Cross-modal Mutual Knowledge Transfer](https://mlsanity.com/paper/2210.14823/) (2022)
- [Bias-Conflict Sample Synthesis and Adversarial Removal Debias Strategy for Temporal Sentence Grounding in Video](https://mlsanity.com/paper/2401.07567/) (2024)
- [Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language](https://mlsanity.com/paper/2605.29793/) (2026)
- [EtC: Temporal Boundary Expand then Clarify for Weakly Supervised Video Grounding with Multimodal Large Language Model](https://mlsanity.com/paper/2312.02483/) (2023)
- [Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding](https://mlsanity.com/paper/2204.01450/) (2022)
- [Visual Self-paced Iterative Learning for Unsupervised Temporal Action Localization](https://mlsanity.com/paper/2312.07384/) (2023)
- [Learning to Unify Audio, Visual and Text for Audio-Enhanced Multilingual Visual Answer Localization](https://mlsanity.com/paper/2411.02851/) (2024)
- [VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval](https://mlsanity.com/paper/2412.01558/) (2024)
- [Effective and Robust Adversarial Training against Data and Label Corruptions](https://mlsanity.com/paper/2405.04191/) (2024)
- [Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding](https://mlsanity.com/paper/2605.30742/) (2026)
- [Exploring Iterative Refinement with Diffusion Models for Video Grounding](https://mlsanity.com/paper/2310.17189/) (2023)
- [CoSTL: Comprehensive Spatial-Temporal Representation Learning for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2606.01149/) (2026)
- [Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video Localization](https://mlsanity.com/paper/2401.08232/) (2024)
- [LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization](https://mlsanity.com/paper/2505.24282/) (2025)
- [CACR:Reinforcing Temporal Answer Grounding in Instructional Video via Candidate-Aware Causal Reasoning](https://mlsanity.com/paper/2606.08436/) (2026)

## Abstract (arXiv)

Video moment retrieval targets at retrieving a moment in a video for a given language query. The challenges of this task include 1) the requirement of localizing the relevant moment in an untrimmed video, and 2) bridging the semantic gap between textual query and video contents. To tackle those problems, early approaches adopt the sliding window or uniform sampling to collect video clips first and then match each clip with the query. Obviously, these strategies are time-consuming and often lead to unsatisfied accuracy in localization due to the unpredictable length of the golden moment. To avoid the limitations, researchers recently attempt to directly predict the relevant moment boundaries without the requirement to generate video clips first. One mainstream approach is to generate a multimodal feature vector for the target query and video frames (e.g., concatenation) and then use a regression approach upon the multimodal feature vector for boundary detection. Although some progress has been achieved by this approach, we argue that those methods have not well captured the cross-modal interactions between the query and video frames. In this paper, we propose an Attentive Cross-modal Relevance Matching (ACRM) model which predicts the temporal boundaries based on an interaction modeling. In addition, an attention module is introduced to assign higher weights to query words with richer semantic cues, which are considered to be more important for finding relevant video contents. Another contribution is that we propose an additional predictor to utilize the internal frames in the model training to improve the localization accuracy. Extensive experiments on two datasets TACoS and Charades-STA demonstrate the superiority of our method over several state-of-the-art methods. Ablation studies have been also conducted to examine the effectiveness of different modules in our ACRM model.