# DiVR: incorporating context from diverse VR scenes for human trajectory prediction

arXiv: https://arxiv.org/abs/2411.08409 · HTML: https://mlsanity.com/paper/2411.08409/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs](https://mlsanity.com/paper/1704.02901/) (2017)
- [Molecular Graph Convolutions: Moving Beyond Fingerprints](https://mlsanity.com/paper/1603.00856/) (2016)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [A Spatio-temporal Transformer for 3D Human Motion Prediction](https://mlsanity.com/paper/2004.08692/) (2020)
- [Multi-Agent Tensor Fusion for Contextual Trajectory Prediction](https://mlsanity.com/paper/1904.04776/) (2019)
- [Back to MLP: A Simple Baseline for Human Motion Prediction](https://mlsanity.com/paper/2207.01567/) (2022)
- [Context-aware Human Motion Prediction](https://mlsanity.com/paper/1904.03419/) (2019)
- [GIMO: Gaze-Informed Human Motion Prediction in Context](https://mlsanity.com/paper/2204.09443/) (2022)
- [Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphs](https://mlsanity.com/paper/2112.04222/) (2021)
- [Towards Explainable Motion Prediction using Heterogeneous Graph Representations](https://mlsanity.com/paper/2212.03806/) (2022)
- [CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations](https://mlsanity.com/paper/2109.00181/) (2021)
- [Relation-based Motion Prediction using Traffic Scene Graphs](https://mlsanity.com/paper/2212.02503/) (2022)

## Abstract (arXiv)

Virtual environments provide a rich and controlled setting for collecting detailed data on human behavior, offering unique opportunities for predicting human trajectories in dynamic scenes. However, most existing approaches have overlooked the potential of these environments, focusing instead on static contexts without considering userspecific factors. Employing the CREATTIVE3D dataset, our work models trajectories recorded in virtual reality (VR) scenes for diverse situations including road-crossing tasks with user interactions and simulated visual impairments. We propose Diverse Context VR Human Motion Prediction (DiVR), a cross-modal transformer based on the Perceiver architecture that integrates both static and dynamic scene context using a heterogeneous graph convolution network. We conduct extensive experiments comparing DiVR against existing architectures including MLP, LSTM, and transformers with gaze and point cloud context. Additionally, we also stress test our model's generalizability across different users, tasks, and scenes. Results show that DiVR achieves higher accuracy and adaptability compared to other models and to static graphs. This work highlights the advantages of using VR datasets for context-aware human trajectory modeling, with potential applications in enhancing user experiences in the metaverse. Our source code is publicly available at https://gitlab.inria.fr/ffrancog/creattive3d-divr-model.