# Active Learning of Ordinal Embeddings: A User Study on Football Data

arXiv: https://arxiv.org/abs/2207.12710 · HTML: https://mlsanity.com/paper/2207.12710/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 43 references that are in the map.
- First posted on arXiv 26 Jul 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [Improved Embeddings with Easy Positive Triplet Mining](https://mlsanity.com/paper/1904.04370/) (2019)
- [Cost-Effective HITs for Relative Similarity Comparisons](https://mlsanity.com/paper/1404.3291/) (2014)
- [Active embedding search via noisy paired comparisons](https://mlsanity.com/paper/1905.04363/) (2019)
- [Fine-Grained Retrieval of Sports Plays using Tree-Based Alignment of Trajectories](https://mlsanity.com/paper/1710.02255/) (2017)
- [Active Ordinal Querying for Tuplewise Similarity Learning](https://mlsanity.com/paper/1910.04115/) (2019)
- [Uncertainty Estimates for Ordinal Embeddings](https://mlsanity.com/paper/1906.11655/) (2019)
- [Active metric learning and classification using similarity queries](https://mlsanity.com/paper/2202.01953/) (2022)
- [Retrieving Similar Trajectories from Cellular Data at City Scale](https://mlsanity.com/paper/1907.12371/) (2019)

## Built on by (most cited first, 1 of 1)

- [Effective Data Pruning through Score Extrapolation](https://mlsanity.com/paper/2506.09010/) (2025)

## Abstract (arXiv)

Humans innately measure distance between instances in an unlabeled dataset using an unknown similarity function. Distance metrics can only serve as proxy for similarity in information retrieval of similar instances. Learning a good similarity function from human annotations improves the quality of retrievals. This work uses deep metric learning to learn these user-defined similarity functions from few annotations for a large football trajectory dataset. We adapt an entropy-based active learning method with recent work from triplet mining to collect easy-to-answer but still informative annotations from human participants and use them to train a deep convolutional network that generalizes to unseen samples. Our user study shows that our approach improves the quality of the information retrieval compared to a previous deep metric learning approach that relies on a Siamese network. Specifically, we shed light on the strengths and weaknesses of passive sampling heuristics and active learners alike by analyzing the participants' response efficacy. To this end, we collect accuracy, algorithmic time complexity, the participants' fatigue and time-to-response, qualitative self-assessment and statements, as well as the effects of mixed-expertise annotators and their consistency on model performance and transfer-learning.