# PathFinder: Attention-Driven Dynamic Non-Line-of-Sight Tracking with a Mobile Robot

arXiv: https://arxiv.org/abs/2404.05024 · HTML: https://mlsanity.com/paper/2404.05024/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 7 Apr 2024.

## Builds on (most cited first, 18 of 18)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://mlsanity.com/paper/2307.06304/) (2023)
- [Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar](https://mlsanity.com/paper/1912.06613/) (2019)
- [Robust Stereo Visual Inertial Odometry for Fast Autonomous Flight](https://mlsanity.com/paper/1712.00036/) (2017)
- [Recent Advances in Imaging Around Corners](https://mlsanity.com/paper/1910.05613/) (2019)
- [Steady-state Non-Line-of-Sight Imaging](https://mlsanity.com/paper/1811.09910/) (2018)
- [Computational Mirrors: Blind Inverse Light Transport by Deep Matrix Factorization](https://mlsanity.com/paper/1912.02314/) (2019)
- [Two-Dimensional Non-Line-of-Sight Scene Estimation from a Single Edge Occluder](https://mlsanity.com/paper/2006.09241/) (2020)
- [Recent Advances on Non-Line-of-Sight Imaging: Conventional Physical Models, Deep Learning, and New Scenes](https://mlsanity.com/paper/2104.13807/) (2021)
- [What You Can Learn by Staring at a Blank Wall](https://mlsanity.com/paper/2108.13027/) (2021)
- [Optically lightweight tracking of objects around a corner](https://mlsanity.com/paper/1606.01873/) (2016)
- [PlaneRecNet: Multi-Task Learning with Cross-Task Consistency for Piece-Wise Plane Detection and Reconstruction from a Single RGB Image](https://mlsanity.com/paper/2110.11219/) (2021)
- [Flash Photography for Data-Driven Hidden Scene Recovery](https://mlsanity.com/paper/1810.11710/) (2018)
- [Adaptive Lighting for Data-Driven Non-Line-of-Sight 3D Localization and Object Identification](https://mlsanity.com/paper/1905.11595/) (2019)
- [Multi-modal Non-line-of-sight Passive Imaging](https://mlsanity.com/paper/1807.02444/) (2018)
- [Propagate And Calibrate: Real-time Passive Non-line-of-sight Tracking](https://mlsanity.com/paper/2303.11791/) (2023)

## Abstract (arXiv)

The study of non-line-of-sight (NLOS) imaging is growing due to its many potential applications, including rescue operations and pedestrian detection by self-driving cars. However, implementing NLOS imaging on a moving camera remains an open area of research. Existing NLOS imaging methods rely on time-resolved detectors and laser configurations that require precise optical alignment, making it difficult to deploy them in dynamic environments. This work proposes a data-driven approach to NLOS imaging, PathFinder, that can be used with a standard RGB camera mounted on a small, power-constrained mobile robot, such as an aerial drone. Our experimental pipeline is designed to accurately estimate the 2D trajectory of a person who moves in a Manhattan-world environment while remaining hidden from the camera's field-of-view. We introduce a novel approach to process a sequence of dynamic successive frames in a line-of-sight (LOS) video using an attention-based neural network that performs inference in real-time. The method also includes a preprocessing selection metric that analyzes images from a moving camera which contain multiple vertical planar surfaces, such as walls and building facades, and extracts planes that return maximum NLOS information. We validate the approach on in-the-wild scenes using a drone for video capture, thus demonstrating low-cost NLOS imaging in dynamic capture environments.