# Moment of Untruth: Dealing with Negative Queries in Video Moment Retrieval

arXiv: https://arxiv.org/abs/2502.08544 · HTML: https://mlsanity.com/paper/2502.08544/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 12 Feb 2025.

## Builds on (most cited first, 30 of 34)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [Span-based Localizing Network for Natural Language Video Localization](https://mlsanity.com/paper/2004.13931/) (2020)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [UniVTG: Towards Unified Video-Language Temporal Grounding](https://mlsanity.com/paper/2307.16715/) (2023)
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2303.13874/) (2023)
- [To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression](https://mlsanity.com/paper/1804.07014/) (2018)
- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [Multilevel Language and Vision Integration for Text-to-Clip Retrieval](https://mlsanity.com/paper/1804.05113/) (2018)
- [Context-aware Biaffine Localizing Network for Temporal Sentence Grounding](https://mlsanity.com/paper/2103.11555/) (2021)
- [Interventional Video Grounding with Dual Contrastive Learning](https://mlsanity.com/paper/2106.11013/) (2021)
- [MAC: Mining Activity Concepts for Language-based Temporal Localization](https://mlsanity.com/paper/1811.08925/) (2018)
- [Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction](https://mlsanity.com/paper/1909.05010/) (2019)
- [Proposal-free Temporal Moment Localization of a Natural-Language Query in Video using Guided Attention](https://mlsanity.com/paper/1908.07236/) (2019)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries](https://mlsanity.com/paper/2107.09609/) (2021)
- [Optimal strategies for reject option classifiers](https://mlsanity.com/paper/2101.12523/) (2021)
- [Video Corpus Moment Retrieval with Contrastive Learning](https://mlsanity.com/paper/2105.06247/) (2021)
- [TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2401.02309/) (2024)
- [Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos](https://mlsanity.com/paper/2007.14164/) (2020)
- [Natural Language Video Localization: A Revisit in Span-based Question Answering Framework](https://mlsanity.com/paper/2102.13558/) (2021)
- [On Pursuit of Designing Multi-modal Transformer for Video Grounding](https://mlsanity.com/paper/2109.06085/) (2021)
- [End-to-end Multi-modal Video Temporal Grounding](https://mlsanity.com/paper/2107.05624/) (2021)
- [Fine-grained Iterative Attention Network for TemporalLanguage Localization in Videos](https://mlsanity.com/paper/2008.02448/) (2020)
- [Memory-Guided Semantic Learning Network for Temporal Sentence Grounding](https://mlsanity.com/paper/2201.00454/) (2022)
- [MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer](https://mlsanity.com/paper/2305.00355/) (2023)
- [R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding](https://mlsanity.com/paper/2404.00801/) (2024)

## Built on by (most cited first, 10 of 10)

- [Tempo-R0: A Video-MLLM for Temporal Video Grounding through Efficient Temporal Sensing Reinforcement Learning](https://mlsanity.com/paper/2507.04702/) (2025)
- [Learning to Refuse: Refusal-Aware Reinforcement Fine-Tuning for Hard-Irrelevant Queries in Video Temporal Grounding](https://mlsanity.com/paper/2511.23151/) (2025)
- [Conditional Multi-Event Temporal Grounding in Long-Form Video](https://mlsanity.com/paper/2606.15320/) (2026)
- [LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization](https://mlsanity.com/paper/2505.24282/) (2025)
- [Beyond Caption-Based Queries for Video Moment Retrieval](https://mlsanity.com/paper/2603.02363/) (2026)
- [Retrieving Any Relevant Moments: Benchmark and Models for Generalized Moment Retrieval](https://mlsanity.com/paper/2605.02623/) (2026)
- [Does This Moment Justify the Recommendation? Counterfactual Behavior-Grounded Evidence Retrieval for Personalized Video Recommendation](https://mlsanity.com/paper/2609.00996/) (2026)
- [IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding](https://mlsanity.com/paper/2609.09736/) (2026)
- [Grounding with Confidence: Controllable Generative Video Temporal Grounding](https://mlsanity.com/paper/2609.39883/) (2026)
- [Investigating Query-Insensitive Behavior in Spatio-Temporal Video Grounding](https://mlsanity.com/paper/2610.06018/) (2026)

## Abstract (arXiv)

Video Moment Retrieval is a common task to evaluate the performance of visual-language models - it involves localising start and end times of moments in videos from query sentences. The current task formulation assumes that the queried moment is present in the video, resulting in false positive moment predictions when irrelevant query sentences are provided. In this paper we propose the task of Negative-Aware Video Moment Retrieval (NA-VMR), which considers both moment retrieval accuracy and negative query rejection accuracy. We make the distinction between In-Domain and Out-of-Domain negative queries and provide new evaluation benchmarks for two popular video moment retrieval datasets: QVHighlights and Charades-STA. We analyse the ability of current SOTA video moment retrieval approaches to adapt to Negative-Aware Video Moment Retrieval and propose UniVTG-NA, an adaptation of UniVTG designed to tackle NA-VMR. UniVTG-NA achieves high negative rejection accuracy (avg. 98.4%) scores while retaining moment retrieval scores to within 3.87% Recall@1. Dataset splits and code are available at https://github.com/keflanagan/MomentofUntruth