# Single-Stage Visual Query Localization in Egocentric Videos

arXiv: https://arxiv.org/abs/2306.09324 · HTML: https://mlsanity.com/paper/2306.09324/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 60 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Local-Global Video-Text Interactions for Temporal Grounding](https://mlsanity.com/paper/2004.07514/) (2020)
- [Siam R-CNN: Visual Tracking by Re-Detection](https://mlsanity.com/paper/1911.12836/) (2019)

## Built on by (most cited first, 20 of 20)

- [OmniGlue: Generalizable Feature Matching with Foundation Model Guidance](https://mlsanity.com/paper/2405.12979/) (2024)
- [V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning](https://mlsanity.com/paper/2404.12353/) (2024)
- [Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding](https://mlsanity.com/paper/2503.09143/) (2025)
- [Text-Conditioned Resampler For Long Form Video Understanding](https://mlsanity.com/paper/2312.11897/) (2023)
- [Challenges and Trends in Egocentric Vision: A Survey](https://mlsanity.com/paper/2503.15275/) (2025)
- [Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding](https://mlsanity.com/paper/2403.16276/) (2024)
- [Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory](https://mlsanity.com/paper/2411.16934/) (2024)
- [RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations](https://mlsanity.com/paper/2412.01826/) (2024)
- [PRVQL: Progressive Knowledge-guided Refinement for Robust Egocentric Visual Query Localization](https://mlsanity.com/paper/2502.07707/) (2025)
- [Localizing Events in Videos with Multimodal Queries](https://mlsanity.com/paper/2406.10079/) (2024)
- [HANDI: Hand-Centric Text-and-Image Conditioned Video Generation](https://mlsanity.com/paper/2412.04189/) (2024)
- [REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders](https://mlsanity.com/paper/2505.18153/) (2025)
- [EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision](https://mlsanity.com/paper/2605.31557/) (2026)
- [EgoSplat: Open-Vocabulary Egocentric Scene Understanding with Language Embedded 3D Gaussian Splatting](https://mlsanity.com/paper/2503.11345/) (2025)
- [PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videos](https://mlsanity.com/paper/2407.09503/) (2024)
- [EAGLE: Episodic Appearance- and Geometry-aware Memory for Unified 2D-3D Visual Query Localization in Egocentric Vision](https://mlsanity.com/paper/2511.08007/) (2025)
- [HERO-VQL: Hierarchical, Egocentric and Robust Visual Query Localization](https://mlsanity.com/paper/2509.00385/) (2025)
- [Towards Visual Query Localization in the 3D World](https://mlsanity.com/paper/2605.01498/) (2026)
- [Towards Visual Query Segmentation in the Wild](https://mlsanity.com/paper/2603.08898/) (2026)
- [PGL-3D: Towards Progressive Geometric Learning for 3D Visual Query Localization](https://mlsanity.com/paper/2609.33558/) (2026)

## Abstract (arXiv)

Visual Query Localization on long-form egocentric videos requires spatio-temporal search and localization of visually specified objects and is vital to build episodic memory systems. Prior work develops complex multi-stage pipelines that leverage well-established object detection and tracking methods to perform VQL. However, each stage is independently trained and the complexity of the pipeline results in slow inference speeds. We propose VQLoC, a novel single-stage VQL framework that is end-to-end trainable. Our key idea is to first build a holistic understanding of the query-video relationship and then perform spatio-temporal localization in a single shot manner. Specifically, we establish the query-video relationship by jointly considering query-to-frame correspondences between the query and each video frame and frame-to-frame correspondences between nearby video frames. Our experiments demonstrate that our approach outperforms prior VQL methods by 20% accuracy while obtaining a 10x improvement in inference speed. VQLoC is also the top entry on the Ego4D VQ2D challenge leaderboard. Project page: https://hwjiang1510.github.io/VQLoC/