# Eye Gaze-Informed and Context-Aware Pedestrian Trajectory Prediction in Shared Spaces with Automated Shuttles: A Virtual Reality Study

arXiv: https://arxiv.org/abs/2603.19812 · HTML: https://mlsanity.com/paper/2603.19812/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 124 references that are in the map.
- First posted on arXiv 20 Mar 2026.

## Builds on (most cited first, 27 of 27)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset](https://mlsanity.com/paper/2104.10133/) (2021)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoning](https://mlsanity.com/paper/2204.08770/) (2022)
- [Autonomous Vehicles that Interact with Pedestrians: A Survey of Theory and Practice](https://mlsanity.com/paper/1805.11773/) (2018)
- [Collaborative Motion Prediction via Neural Motion Message Passing](https://mlsanity.com/paper/2003.06594/) (2020)
- [Top-view Trajectories: A Pedestrian Dataset of Vehicle-Crowd Interaction from Controlled Experiments and Crowded Campus](https://mlsanity.com/paper/1902.00487/) (2019)
- [Pedestrian Trajectory Prediction in Pedestrian-Vehicle Mixed Environments: A Systematic Review](https://mlsanity.com/paper/2308.06419/) (2023)
- [MX-LSTM: mixing tracklets and vislets to jointly forecast trajectories and head poses](https://mlsanity.com/paper/1805.00652/) (2018)
- [Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting](https://mlsanity.com/paper/2211.03165/) (2022)
- [Pedestrian Behavior Prediction for Automated Driving: Requirements, Metrics, and Relevant Features](https://mlsanity.com/paper/2012.08418/) (2020)
- [A context-aware pedestrian trajectory prediction framework for automated vehicles](https://mlsanity.com/paper/2104.08123/) (2021)
- [SocialCVAE: Predicting Pedestrian Trajectory via Interaction Conditioned Latents](https://mlsanity.com/paper/2402.17339/) (2024)
- [Cross or Wait? Predicting Pedestrian Interaction Outcomes at Unsignalized Crossings](https://mlsanity.com/paper/2304.08260/) (2023)
- [Do Pedestrians Pay Attention? Eye Contact Detection in the Wild](https://mlsanity.com/paper/2112.04212/) (2021)
- [Understanding Pedestrian-Vehicle Interactions with Vehicle Mounted Vision: An LSTM Model and Empirical Analysis](https://mlsanity.com/paper/1905.05350/) (2019)
- [Off The Beaten Sidewalk: Pedestrian Prediction In Shared Spaces For Autonomous Vehicles](https://mlsanity.com/paper/2006.00962/) (2020)
- [Pedestrian-Vehicle Interaction in Shared Space: Insights for Autonomous Vehicles](https://mlsanity.com/paper/2403.04933/) (2024)
- [Context-aware Multi-task Learning for Pedestrian Intent and Trajectory Prediction](https://mlsanity.com/paper/2407.17162/) (2024)
- [Stochastic Trajectory Prediction under Unstructured Constraints](https://mlsanity.com/paper/2503.14203/) (2025)
- [Effect of eHMI on pedestrian road crossing behavior in shared space with Automated Vehicles-A Virtual Reality study](https://mlsanity.com/paper/2308.05654/) (2023)
- [Modeling Eye Gaze Velocity Trajectories using GANs with Spectral Loss for Enhanced Fidelity](https://mlsanity.com/paper/2412.04184/) (2024)
- [Impact of Smartphone Distraction on Pedestrians' Crossing Behaviour: An Application of Head-Mounted Immersive Virtual Reality](https://mlsanity.com/paper/1806.06454/) (2018)
- [F-RDW: Redirected Walking with Forecasting Future Position](https://mlsanity.com/paper/2304.03497/) (2023)
- [Crowdsourcing eHMI Designs: A Participatory Approach to Autonomous Vehicle-Pedestrian Communication](https://mlsanity.com/paper/2506.18605/) (2025)

## Built on by (most cited first, 1 of 1)

- [Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models](https://mlsanity.com/paper/2606.09142/) (2026)

## Abstract (arXiv)

To address this gap, we conduct a Virtual Reality experiment in which pedestrians interact with automated shuttles under varying approach angles (45deg, 90deg, 135deg) and continuous-traffic conditions (single shuttle, two shuttles with 3 or 5-second gaps), collecting synchronized motion, eye gaze, and head orientation data. To investigate to what extent, under what conditions, and in what form fine-grained eye gaze is informative for pedestrian motion prediction, we develop a multi-modal prediction model that fuses these signals through modality-specific encoders, and systematically ablate gaze representations against head orientation and situational context. We report three main results. First, the predictive value of eye gaze is angle-dependent and tightly coupled with eye-head-body coordination: at acute angles where pedestrians actively redirect gaze to acquire the shuttle, eye gaze carries information that head orientation alone misses. Second, continuous gaze orientation outperforms categorical semantic fixation labels, with the optimal encoding frame (global or body-relative) depending on whether gaze is used alone or jointly with context. Third, eye gaze and situational context provide complementary predictive information: their combination reduces final displacement error (FDE) by 8.47%, close to the sum of their individual contributions. Together, these findings highlight the value of incorporating human perceptual signals into pedestrian behavior prediction and motivate a human-centered complement to vehicle-centric modeling approaches. Our code is available at https://github.com/danyayay/GazeX.git.