# Towards Visual Query Localization in the 3D World

arXiv: https://arxiv.org/abs/2605.01498 · HTML: https://mlsanity.com/paper/2605.01498/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 2 May 2026.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PointPainting: Sequential Fusion for 3D Object Detection](https://mlsanity.com/paper/1911.10150/) (2019)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI](https://mlsanity.com/paper/2312.16170/) (2023)
- [Focal Sparse Convolutional Networks for 3D Object Detection](https://mlsanity.com/paper/2204.12463/) (2022)
- [Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction](https://mlsanity.com/paper/1904.01690/) (2019)
- [TAPIP3D: Tracking Any Point in Persistent 3D Geometry](https://mlsanity.com/paper/2504.14717/) (2025)
- [Where is my Wallet? Modeling Object Proposal Sets for Egocentric Visual Query Localization](https://mlsanity.com/paper/2211.10528/) (2022)
- [MBPTrack: Improving 3D Point Cloud Tracking with Memory Networks and Box Priors](https://mlsanity.com/paper/2303.05071/) (2023)
- [Single-Stage Visual Query Localization in Egocentric Videos](https://mlsanity.com/paper/2306.09324/) (2023)
- [LiDAR-Camera Calibration using 3D-3D Point correspondences](https://mlsanity.com/paper/1705.09785/) (2017)
- [Negative Frames Matter in Egocentric Visual Query 2D Localization](https://mlsanity.com/paper/2208.01949/) (2022)
- [RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations](https://mlsanity.com/paper/2412.01826/) (2024)
- [PRVQL: Progressive Knowledge-guided Refinement for Robust Egocentric Visual Query Localization](https://mlsanity.com/paper/2502.07707/) (2025)
- [GSOT3D: Towards Generic 3D Single Object Tracking in the Wild](https://mlsanity.com/paper/2412.02129/) (2024)

## Built on by (most cited first, 1 of 1)

- [PGL-3D: Towards Progressive Geometric Learning for 3D Visual Query Localization](https://mlsanity.com/paper/2609.33558/) (2026)

## Abstract (arXiv)

Visual query localization (VQL) aims to predict the spatio-temporal response of the most recent occurrence in a sequence given a query. Currently, most research focuses on visual query localization in 2D videos, while its counterpart in 3D space has received little attention. In this paper, we make the first attempt to address visual query localization in the 3D world by introducing a novel benchmark, dubbed 3DVQL. Specifically, 3DVQL contains 2,002 sequences with around 170,000 frames and 6.4K response track segments from 38 object categories. Each sequence in 3DVQL is provided with multiple modalities, including point clouds, RGB images, and depth images, to support flexible research. To ensure high-quality annotations, each sequence is manually annotated with multiple rounds of verification and refinement. To the best of our knowledge, 3DVQL is the first benchmark for 3D multimodal visual query localization. To facilitate comparison in subsequent research, we implement a series of representative 3D multimodal VQL baselines using point clouds and RGB images. The experimental results show that existing methods exhibit significant performance variations across different fusion modules. To encourage future research, we propose a lift-and-attention fusion algorithm named LaF, which significantly outperforms existing baseline models. Our benchmark and model will be publicly released at https://github.com/wuhengliangliang/3DVQL.