# When We First Met: Visual-Inertial Person Localization for Co-Robot Rendezvous

arXiv: https://arxiv.org/abs/2006.09959 · HTML: https://mlsanity.com/paper/2006.09959/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2020.

## Built on by (most cited first, 2 of 2)

- [3DMODT: Attention-Guided Affinities for Joint Detection & Tracking in 3D Point Clouds](https://mlsanity.com/paper/2211.00746/) (2022)
- [MoBind: Motion Binding for Fine-Grained IMU-Video Pose Alignment](https://mlsanity.com/paper/2602.19004/) (2026)

## Abstract (arXiv)

We aim to enable robots to visually localize a target person through the aid of an additional sensing modality -- the target person's 3D inertial measurements. The need for such technology may arise when a robot is to meet person in a crowd for the first time or when an autonomous vehicle must rendezvous with a rider amongst a crowd without knowing the appearance of the person in advance. A person's inertial information can be measured with a wearable device such as a smart-phone and can be shared selectively with an autonomous system during the rendezvous. We propose a method to learn a visual-inertial feature space in which the motion of a person in video can be easily matched to the motion measured by a wearable inertial measurement unit (IMU). The transformation of the two modalities into the joint feature space is learned through the use of a contrastive loss which forces inertial motion features and video motion features generated by the same person to lie close in the joint feature space. To validate our approach, we compose a dataset of over 60,000 video segments of moving people along with wearable IMU data. Our experiments show that our proposed method is able to accurately localize a target person with 80.7% accuracy using only 5 seconds of IMU data and video.