# Is First Person Vision Challenging for Object Tracking?

arXiv: https://arxiv.org/abs/2108.13665 · HTML: https://mlsanity.com/paper/2108.13665/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 110 references that are in the map.
- First posted on arXiv 31 Aug 2021.

## Builds on (most cited first, 30 of 62)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [ECO: Efficient Convolution Operators for Tracking](https://mlsanity.com/paper/1611.09224/) (2016)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [Probabilistic Regression for Visual Tracking](https://mlsanity.com/paper/2003.12565/) (2020)
- [Ocean: Object-aware Anchor-free Tracking](https://mlsanity.com/paper/2006.10721/) (2020)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Siamese Box Adaptive Network for Visual Tracking](https://mlsanity.com/paper/2003.06761/) (2020)
- [SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines](https://mlsanity.com/paper/1911.06188/) (2019)
- [Deeper and Wider Siamese Networks for Real-Time Visual Tracking](https://mlsanity.com/paper/1901.01660/) (2019)
- [Reconstructing Hand-Object Interactions in the Wild](https://mlsanity.com/paper/2012.09856/) (2020)
- [Need for Speed: A Benchmark for Higher Frame Rate Object Tracking](https://mlsanity.com/paper/1703.05884/) (2017)
- [MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking](https://mlsanity.com/paper/2010.07548/) (2020)
- [Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video](https://mlsanity.com/paper/2005.02190/) (2020)
- [Know Your Surroundings: Exploiting Scene Information for Object Tracking](https://mlsanity.com/paper/2003.11014/) (2020)
- [A Novel Performance Evaluation Methodology for Single-Target Trackers](https://mlsanity.com/paper/1503.01313/) (2015)
- [What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention](https://mlsanity.com/paper/1905.09035/) (2019)
- [Learning to Track at 100 FPS with Deep Regression Networks](https://mlsanity.com/paper/1604.01802/) (2016)
- [Learning What to Learn for Video Object Segmentation](https://mlsanity.com/paper/2003.11540/) (2020)

## Built on by (most cited first, 21 of 21)

- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [EgoLife: Towards Egocentric Life Assistant](https://mlsanity.com/paper/2503.03803/) (2025)
- [MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain](https://mlsanity.com/paper/2209.08691/) (2022)
- [Egocentric Prediction of Action Target in 3D](https://mlsanity.com/paper/2203.13116/) (2022)
- [EgoTracks: A Long-term Egocentric Visual Object Tracking Dataset](https://mlsanity.com/paper/2301.03213/) (2023)
- [Instance Tracking in 3D Scenes from Egocentric Videos](https://mlsanity.com/paper/2312.04117/) (2023)
- [360VOT: A New Benchmark Dataset for Omnidirectional Visual Object Tracking](https://mlsanity.com/paper/2307.14630/) (2023)
- [Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory](https://mlsanity.com/paper/2411.16934/) (2024)
- [Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge](https://mlsanity.com/paper/2310.15066/) (2023)
- [360VOTS: Visual Object Tracking and Segmentation in Omnidirectional Videos](https://mlsanity.com/paper/2404.13953/) (2024)
- [Ego3DT: Tracking Every 3D Object in Ego-centric Videos](https://mlsanity.com/paper/2410.08530/) (2024)
- [Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitations](https://mlsanity.com/paper/2311.17938/) (2023)
- [Tracking Skiers from the Top to the Bottom](https://mlsanity.com/paper/2312.09723/) (2023)
- [PVT++: A Simple End-to-End Latency-Aware Visual Tracking Framework](https://mlsanity.com/paper/2211.11629/) (2022)
- [CoCoLoT: Combining Complementary Trackers in Long-Term Visual Tracking](https://mlsanity.com/paper/2205.04261/) (2022)
- [HERO-VQL: Hierarchical, Egocentric and Robust Visual Query Localization](https://mlsanity.com/paper/2509.00385/) (2025)
- [Visualizing Skiers' Trajectories in Monocular Videos](https://mlsanity.com/paper/2304.02994/) (2023)
- [ZJU ReLER Submission for EPIC-KITCHEN Challenge 2023: TREK-150 Single Object Tracking](https://mlsanity.com/paper/2307.02508/) (2023)
- [Is Tracking really more challenging in First Person Egocentric Vision?](https://mlsanity.com/paper/2507.16015/) (2025)
- [Bayesian Decision Making to Localize Visual Queries in 2D](https://mlsanity.com/paper/2305.17611/) (2023)
- [EXOT: Exit-aware Object Tracker for Safe Robotic Manipulation of Moving Object](https://mlsanity.com/paper/2306.05262/) (2023)

## Abstract (arXiv)

Understanding human-object interactions is fundamental in First Person Vision (FPV). Tracking algorithms which follow the objects manipulated by the camera wearer can provide useful cues to effectively model such interactions. Visual tracking solutions available in the computer vision literature have significantly improved their performance in the last years for a large variety of target objects and tracking scenarios. However, despite a few previous attempts to exploit trackers in FPV applications, a methodical analysis of the performance of state-of-the-art trackers in this domain is still missing. In this paper, we fill the gap by presenting the first systematic study of object tracking in FPV. Our study extensively analyses the performance of recent visual trackers and baseline FPV trackers with respect to different aspects and considering a new performance measure. This is achieved through TREK-150, a novel benchmark dataset composed of 150 densely annotated video sequences. Our results show that object tracking in FPV is challenging, which suggests that more research efforts should be devoted to this problem so that tracking could benefit FPV tasks.