# A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus

arXiv: https://arxiv.org/abs/2011.09046 · HTML: https://mlsanity.com/paper/2011.09046/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2020.

## Built on by (most cited first, 26 of 26)

- [QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries](https://mlsanity.com/paper/2107.09609/) (2021)
- [UnLoc: A Unified Framework for Video Localization Tasks](https://mlsanity.com/paper/2308.11062/) (2023)
- [Video Corpus Moment Retrieval with Contrastive Learning](https://mlsanity.com/paper/2105.06247/) (2021)
- [Temporal Sentence Grounding in Videos: A Survey and Future Directions](https://mlsanity.com/paper/2201.08071/) (2022)
- [CONQUER: Contextual Query-aware Ranking for Video Corpus Moment Retrieval](https://mlsanity.com/paper/2109.10016/) (2021)
- [A Survey on Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2109.08039/) (2021)
- [GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval](https://mlsanity.com/paper/2310.05195/) (2023)
- [Selective Query-guided Debiasing for Video Corpus Moment Retrieval](https://mlsanity.com/paper/2210.08714/) (2022)
- [Boundary-aware Self-supervised Learning for Video Scene Segmentation](https://mlsanity.com/paper/2201.05277/) (2022)
- [HLFormer: Enhancing Partially Relevant Video Retrieval with Hyperbolic Learning](https://mlsanity.com/paper/2507.17402/) (2025)
- [Cross-Architecture Self-supervised Video Representation Learning](https://mlsanity.com/paper/2205.13313/) (2022)
- [GMMFormer v2: An Uncertainty-aware Framework for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2405.13824/) (2024)
- [Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2506.07471/) (2025)
- [PRVR: Partially Relevant Video Retrieval](https://mlsanity.com/paper/2208.12510/) (2022)
- [Modal-specific Pseudo Query Generation for Video Corpus Moment Retrieval](https://mlsanity.com/paper/2210.12617/) (2022)
- [Improving Video Corpus Moment Retrieval with Partial Relevance Enhancement](https://mlsanity.com/paper/2402.13576/) (2024)
- [Event-aware Video Corpus Moment Retrieval](https://mlsanity.com/paper/2402.13566/) (2024)
- [Dual Learning with Dynamic Knowledge Distillation and Soft Alignment for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2510.12283/) (2025)
- [Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval](https://mlsanity.com/paper/2604.03653/) (2026)
- [Video Moment Retrieval with Text Query Considering Many-to-Many Correspondence Using Potentially Relevant Pair](https://mlsanity.com/paper/2106.13566/) (2021)
- [Towards Efficient and Robust Moment Retrieval System: A Unified Framework for Multi-Granularity Models and Temporal Reranking](https://mlsanity.com/paper/2504.08384/) (2025)
- [MamFusion: Multi-Mamba with Temporal Fusion for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2506.03473/) (2025)
- [Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2605.06083/) (2026)
- [Coarse to Fine: Video Retrieval before Moment Localization](https://mlsanity.com/paper/2110.07201/) (2021)
- [Enhanced Partially Relevant Video Retrieval through Inter- and Intra-Sample Analysis with Coherence Prediction](https://mlsanity.com/paper/2504.19637/) (2025)
- [Multi-video Moment Ranking with Multimodal Clue](https://mlsanity.com/paper/2301.13606/) (2023)

## Abstract (arXiv)

Identifying a short segment in a long video that semantically matches a text query is a challenging task that has important application potentials in language-based video search, browsing, and navigation. Typical retrieval systems respond to a query with either a whole video or a pre-defined video segment, but it is challenging to localize undefined segments in untrimmed and unsegmented videos where exhaustively searching over all possible segments is intractable. The outstanding challenge is that the representation of a video must account for different levels of granularity in the temporal domain. To tackle this problem, we propose the HierArchical Multi-Modal EncodeR (HAMMER) that encodes a video at both the coarse-grained clip level and the fine-grained frame level to extract information at different scales based on multiple subtasks, namely, video retrieval, segment temporal localization, and masked language modeling. We conduct extensive experiments to evaluate our model on moment localization in video corpus on ActivityNet Captions and TVR datasets. Our approach outperforms the previous methods as well as strong baselines, establishing new state-of-the-art for this task.