# Cognitive State Inference from VR Motion via Motion Foundation Model

arXiv: https://arxiv.org/abs/2509.24255 · HTML: https://mlsanity.com/paper/2509.24255/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [MotionGPT: Human Motion as a Foreign Language](https://mlsanity.com/paper/2306.14795/) (2023)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [MotionBERT: A Unified Perspective on Learning Human Motion Representations](https://mlsanity.com/paper/2210.06551/) (2022)
- [Multivariate LSTM-FCNs for Time Series Classification](https://mlsanity.com/paper/1801.04503/) (2018)
- [AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing](https://mlsanity.com/paper/2207.13784/) (2022)
- [BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis](https://mlsanity.com/paper/2304.11118/) (2023)
- [Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling](https://mlsanity.com/paper/2308.08855/) (2023)
- [Unimotion: Unifying 3D Human Motion Synthesis and Understanding](https://mlsanity.com/paper/2409.15904/) (2024)
- [Scaling Large Motion Models with Million-Level Human Motions](https://mlsanity.com/paper/2410.03311/) (2024)
- [Unique Identification of 50,000+ Virtual Reality Users from Head & Hand Motion Data](https://mlsanity.com/paper/2302.08927/) (2023)
- [Human activity recognition based on time series analysis using U-Net](https://mlsanity.com/paper/1809.08113/) (2018)
- [Explainable Depression Detection via Head Motion Patterns](https://mlsanity.com/paper/2307.12241/) (2023)
- [MoFM: A Large-Scale Human Motion Foundation Model](https://mlsanity.com/paper/2502.05432/) (2025)
- [Inferring Private Personal Attributes of Virtual Reality Users from Head and Hand Motion Data](https://mlsanity.com/paper/2305.19198/) (2023)
- [Motion as Emotion: Detecting Affect and Cognitive Load from Free-Hand Gestures in VR](https://mlsanity.com/paper/2409.12921/) (2024)
- [Predicting User Grasp Intentions in Virtual Reality](https://mlsanity.com/paper/2508.16582/) (2025)

## Abstract (arXiv)

As virtual reality (VR) becomes widespread, head and hand motion data captured by consumer systems has become substantially more common. However, the extent of what can be inferred from such motion remains unclear. This paper investigates whether transient cognitive states, specifically confusion, hesitation, and readiness during different stages of decision-making, can be inferred from VR telemetry alone. We introduce a novel dataset of head and hand motion collected during structured decision-making tasks, with frame-level annotations of these states. We evaluate classical machine learning models, temporal neural networks, and motion foundation models under two protocols: (1) future-in-time prediction for the same users, and (2) cross-user generalization to unseen users. We further propose a VR-native motion adapter that maps sparse VR telemetry to representations compatible with motion foundation models pretrained on large-scale full-body motion data, enabling transfer without explicit full-body reconstruction. To our knowledge, this is the first work to adapt a motion foundation model to VR motion for a classification task. Results show that motion-only sensing captures meaningful signals of cognitive states, and that pretrained motion foundation models generalize more effectively than classical and temporal models even with a small dataset of 24 participants. Our approach achieves 82% accuracy, comparable to and sometimes surpassing human observers. These findings suggest that VR motion encodes richer behavioral information than previously assumed and highlight the potential of large-scale motion pretraining for XR applications. We will release the dataset and modeling framework to support future research.