# XR-DT: Extended Reality-Enhanced Digital Twin for Safe Motion Planning via Human-Aware Model Predictive Path Integral Control

arXiv: https://arxiv.org/abs/2512.05270 · HTML: https://mlsanity.com/paper/2512.05270/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 4 Dec 2025.

## Builds on (most cited first, 13 of 13)

- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [EgoMimic: Scaling Imitation Learning via Egocentric Video](https://mlsanity.com/paper/2410.24221/) (2024)
- [Reactive motion planning with probabilistic safety guarantees](https://mlsanity.com/paper/2011.03590/) (2020)
- [Scenario-Based Motion Planning with Bounded Probability of Collision](https://mlsanity.com/paper/2307.01070/) (2023)
- [Risk-Aware Model Predictive Path Integral Control Using Conditional Value-at-Risk](https://mlsanity.com/paper/2209.12842/) (2022)
- [Digital twin in virtual reality for human-vehicle interactions in the context of autonomous driving](https://mlsanity.com/paper/2303.11463/) (2023)
- [Dynamic Risk-Aware MPPI for Mobile Robots in Crowds via Efficient Monte Carlo Approximations](https://mlsanity.com/paper/2506.21205/) (2025)
- [Comparing State-of-the-Art and Emerging Augmented Reality Interfaces for Autonomous Vehicle-to-Pedestrian Communication](https://mlsanity.com/paper/2102.02783/) (2021)
- [Towards Using Fast Embedded Model Predictive Control for Human-Aware Predictive Robot Navigation](https://mlsanity.com/paper/2405.12616/) (2024)
- [Chance-Constrained Sampling-Based MPC for Collision Avoidance in Uncertain Dynamic Environments](https://mlsanity.com/paper/2501.08520/) (2025)

## Abstract (arXiv)

As mobile robots increasingly operate alongside humans in shared workspaces, ensuring safe, efficient, and interpretable Human-Robot Interaction (HRI) has become a pressing challenge. While substantial progress has been devoted to human behavior prediction, limited attention has been paid to how humans perceive, interpret, and trust robots' inferences and how robots plan safe and efficient trajectories based on predicted human behaviors. To address these challenges, this paper presents XR-DT, an eXtended Reality-enhanced Digital Twin framework for mobile robots, which bridges physical and virtual spaces to enable bi-directional understanding between humans and robots. Our hierarchical XR-DT architecture integrates augmented-, virtual-, and mixed-reality layers, fusing real-time sensor data, simulated environments in the Unity game engine, and human feedback captured through wearable XR devices. Within this framework, we design a novel Human-Aware Model Predictive Path Integral (HA-MPPI) control model, an MPPI-based motion planner that incorporates ATLAS (Attention-based Trajectory Learning with Anticipatory Sensing), a multi-modal Transformer model designed for egocentric human trajectory prediction via XR headsets. Extensive real-world experimental results demonstrate accurate human trajectory prediction, and safe and efficient robot navigation, validating the HA-MPPI's effectiveness within the XR-DT framework. By embedding human behavior, environmental dynamics, and robot navigation into the XR-DT framework, our system enables interpretable, trustworthy, and adaptive HRI.