# AxIoU: An Axiomatically Justified Measure for Video Moment Retrieval

arXiv: https://arxiv.org/abs/2203.16062 · HTML: https://mlsanity.com/paper/2203.16062/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language](https://mlsanity.com/paper/1912.03590/) (2019)
- [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://mlsanity.com/paper/2001.09099/) (2020)
- [Dense Regression Network for Video Grounding](https://mlsanity.com/paper/2004.03545/) (2020)
- [To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression](https://mlsanity.com/paper/1804.07014/) (2018)
- [MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment](https://mlsanity.com/paper/1812.00087/) (2018)
- [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/1910.14303/) (2019)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [ExCL: Extractive Clip Localization Using Natural Language Descriptions](https://mlsanity.com/paper/1904.02755/) (2019)
- [Diagnosing Error in Temporal Action Detectors](https://mlsanity.com/paper/1807.10706/) (2018)
- [Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos](https://mlsanity.com/paper/1901.06829/) (2019)
- [Uncovering Hidden Challenges in Query-Based Video Moment Retrieval](https://mlsanity.com/paper/2009.00325/) (2020)
- [Probabilistic Object Detection: Definition and Evaluation](https://mlsanity.com/paper/1811.10800/) (2018)
- [Localization Recall Precision (LRP): A New Performance Metric for Object Detection](https://mlsanity.com/paper/1807.01696/) (2018)
- [What Actions are Needed for Understanding Human Actions in Videos?](https://mlsanity.com/paper/1708.02696/) (2017)
- [Evaluation Measures for Quantification: An Axiomatic Approach](https://mlsanity.com/paper/1809.01991/) (2018)
- [An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric](https://mlsanity.com/paper/1805.02334/) (2018)

## Abstract (arXiv)

Evaluation measures have a crucial impact on the direction of research. Therefore, it is of utmost importance to develop appropriate and reliable evaluation measures for new applications where conventional measures are not well suited. Video Moment Retrieval (VMR) is one such application, and the current practice is to use R@K,θ for evaluating VMR systems. However, this measure has two disadvantages. First, it is rank-insensitive: It ignores the rank positions of successfully localised moments in the top-K ranked list by treating the list as a set. Second, it binarizes the Intersection over Union (IoU) of each retrieved video moment using the threshold θ and thereby ignoring fine-grained localisation quality of ranked moments. We propose an alternative measure for evaluating VMR, called Average Max IoU (AxIoU), which is free from the above two problems. We show that AxIoU satisfies two important axioms for VMR evaluation, namely, Invariance against Redundant Moments and Monotonicity with respect to the Best Moment, and also that R@K,θ satisfies the first axiom only. We also empirically examine how AxIoU agrees with R@K,θ, as well as its stability with respect to change in the test data and human-annotated temporal boundaries.