# Skimming, Locating, then Perusing: A Human-Like Framework for Natural Language Video Localization

arXiv: https://arxiv.org/abs/2207.13450 · HTML: https://mlsanity.com/paper/2207.13450/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 61 references that are in the map.
- First posted on arXiv 27 Jul 2022.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Generation-Augmented Retrieval for Open-domain Question Answering](https://mlsanity.com/paper/2009.08553/) (2020)
- [Dense Regression Network for Video Grounding](https://mlsanity.com/paper/2004.03545/) (2020)
- [To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression](https://mlsanity.com/paper/1804.07014/) (2018)
- [MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment](https://mlsanity.com/paper/1812.00087/) (2018)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Multilevel Language and Vision Integration for Text-to-Clip Retrieval](https://mlsanity.com/paper/1804.05113/) (2018)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [Grounding of Textual Phrases in Images by Reconstruction](https://mlsanity.com/paper/1511.03745/) (2015)
- [Context-aware Biaffine Localizing Network for Temporal Sentence Grounding](https://mlsanity.com/paper/2103.11555/) (2021)
- [Interventional Video Grounding with Dual Contrastive Learning](https://mlsanity.com/paper/2106.11013/) (2021)

## Built on by (most cited first, 21 of 21)

- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos](https://mlsanity.com/paper/2303.07863/) (2023)
- [Hypotheses Tree Building for One-Shot Temporal Sentence Localization](https://mlsanity.com/paper/2301.01871/) (2023)
- [G2L: Semantically Aligned and Uniform Video Grounding via Geodesic and Game Theory](https://mlsanity.com/paper/2307.14277/) (2023)
- [Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding](https://mlsanity.com/paper/2311.03106/) (2023)
- [Rethinking the Video Sampling and Reasoning Strategies for Temporal Sentence Grounding](https://mlsanity.com/paper/2301.00514/) (2023)
- [Unified Static and Dynamic Network: Efficient Temporal Filtering for Video Grounding](https://mlsanity.com/paper/2403.14174/) (2024)
- [Exploiting Auxiliary Caption for Video Grounding](https://mlsanity.com/paper/2301.05997/) (2023)
- [Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding](https://mlsanity.com/paper/2403.11463/) (2024)
- [Rethinking Weakly-supervised Video Temporal Grounding From a Game Perspective](https://mlsanity.com/paper/2605.26441/) (2026)
- [Not All Inputs Are Valid: Towards Open-Set Video Moment Retrieval Using Language](https://mlsanity.com/paper/2605.29812/) (2026)
- [Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language](https://mlsanity.com/paper/2605.29793/) (2026)
- [Jointly Visual- and Semantic-Aware Graph Memory Networks for Temporal Sentence Localization in Videos](https://mlsanity.com/paper/2303.01046/) (2023)
- [SynopGround: A Large-Scale Dataset for Multi-Paragraph Video Grounding from TV Dramas and Synopses](https://mlsanity.com/paper/2408.01669/) (2024)
- [Sim-DETR: Unlock DETR for Temporal Sentence Grounding](https://mlsanity.com/paper/2509.23867/) (2025)
- [Transform-Equivariant Consistency Learning for Temporal Sentence Grounding](https://mlsanity.com/paper/2305.04123/) (2023)
- [Joint Top-Down and Bottom-Up Frameworks for 3D Visual Grounding](https://mlsanity.com/paper/2410.15615/) (2024)
- [Tracking Objects and Activities with Attention for Temporal Sentence Grounding](https://mlsanity.com/paper/2302.10813/) (2023)
- [Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding](https://mlsanity.com/paper/2605.30742/) (2026)
- [LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization](https://mlsanity.com/paper/2505.24282/) (2025)
- [MASRA: MLLM-Assisted Semantic-Relational Consistent Alignment for Video Temporal Grounding](https://mlsanity.com/paper/2605.03398/) (2026)

## Abstract (arXiv)

This paper addresses the problem of natural language video localization (NLVL). Almost all existing works follow the "only look once" framework that exploits a single model to directly capture the complex cross- and self-modal relations among video-query pairs and retrieve the relevant segment. However, we argue that these methods have overlooked two indispensable characteristics of an ideal localization method: 1) Frame-differentiable: considering the imbalance of positive/negative video frames, it is effective to highlight positive frames and weaken negative ones during the localization. 2) Boundary-precise: to predict the exact segment boundary, the model should capture more fine-grained differences between consecutive frames since their variations are often smooth. To this end, inspired by how humans perceive and localize a segment, we propose a two-step human-like framework called Skimming-Locating-Perusing (SLP). SLP consists of a Skimming-and-Locating (SL) module and a Bi-directional Perusing (BP) module. The SL module first refers to the query semantic and selects the best matched frame from the video while filtering out irrelevant frames. Then, the BP module constructs an initial segment based on this frame, and dynamically updates it by exploring its adjacent frames until no frame shares the same activity semantic. Experimental results on three challenging benchmarks show that our SLP is superior to the state-of-the-art methods and localizes more precise segment boundaries.