# Predicting 3D Motion from 2D Video for Behavior-Based VR Biometrics

arXiv: https://arxiv.org/abs/2502.04361 · HTML: https://mlsanity.com/paper/2502.04361/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 54 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose](https://mlsanity.com/paper/2303.07399/) (2023)
- [Digital Transformations of Classrooms in Virtual Reality](https://mlsanity.com/paper/2101.09576/) (2021)
- [Who Is Alyx? A new Behavioral Biometric Dataset for User Identification in XR](https://mlsanity.com/paper/2308.03788/) (2023)
- [Security and Privacy in Virtual Reality: A Literature Survey](https://mlsanity.com/paper/2205.00208/) (2022)
- [Evaluating Deep Networks for Detecting User Familiarity with VR from Hand Interactions](https://mlsanity.com/paper/2401.16443/) (2024)
- [Using Motion Forecasting for Behavior-Based Virtual Reality (VR) Authentication](https://mlsanity.com/paper/2401.16649/) (2024)

## Abstract (arXiv)

Critical VR applications in domains such as healthcare, education, and finance that use traditional credentials, such as PIN, password, or multi-factor authentication, stand the chance of being compromised if a malicious person acquires the user credentials or if the user hands over their credentials to an ally. Recently, a number of approaches on user authentication have emerged that use motions of VR head-mounted displays (HMDs) and hand controllers during user interactions in VR to represent the user's behavior as a VR biometric signature. One of the fundamental limitations of behavior-based approaches is that current on-device tracking for HMDs and controllers lacks capability to perform tracking of full-body joint articulation, losing key signature data encapsulated by the user articulation. In this paper, we propose an approach that uses 2D body joints, namely shoulder, elbow, wrist, hip, knee, and ankle, acquired from the right side of the participants using an external 2D camera. Using a Transformer-based deep neural network, our method uses the 2D data of body joints that are not tracked by the VR device to predict past and future 3D tracks of the right controller, providing the benefit of augmenting 3D knowledge in authentication. Our approach provides a minimum equal error rate (EER) of 0.025, and a maximum EER drop of 0.040 over prior work that uses single-unit 3D trajectory as the input.