# Seeing What the Vehicle Sees: Video-Augmented Virtual Reality for Physical Autonomous Vehicles

arXiv: https://arxiv.org/abs/2609.13224 · HTML: https://mlsanity.com/paper/2609.13224/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Generative simulation for driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Placed in

- End-to-end autonomous driving / [Generative simulation for driving](https://mlsanity.com/driving/generative-simulation-for-driving/)

## Builds on (most cited first, 6 of 6)

- [Explanations in Autonomous Driving: A Survey](https://mlsanity.com/paper/2103.05154/) (2021)
- [Building Trust in Autonomous Vehicles: Role of Virtual Reality Driving Simulators in HMI Design](https://mlsanity.com/paper/2007.13371/) (2020)
- [Realistic pedestrian behaviour in the CARLA simulator using VR and mocap](https://mlsanity.com/paper/2309.04418/) (2023)
- [Mobile Delivery Robots: Mixed Reality-Based Simulation Relying on ROS and Unity 3D](https://mlsanity.com/paper/2006.09002/) (2020)
- [High Speed Emulation in a Vehicle-in-the-Loop Driving Simulator](https://mlsanity.com/paper/2203.03043/) (2022)
- [Portobello: Extending Driving Simulation from the Lab to the Road](https://mlsanity.com/paper/2402.08061/) (2024)

## Abstract (arXiv)

Autonomous vehicles are expected to improve road safety and efficiency, but passengers often remain uncertain about what the vehicle perceives and why it acts as it does. Virtual reality (VR) offers a safe and repeatable medium for presenting this information, yet most passenger-facing VR studies rely on fully simulated vehicles or pre-scripted scenarios, so the motion and perception shown to the user do not originate from a physically operating autonomous system. This paper presents a video-augmented VR framework that couples a physical ROS 2 autonomous robot vehicle to a Unity 6 application deployed on a Meta Quest 3S headset. The vehicle state and live onboard camera stream are transmitted over two independent communication channels, allowing the virtual vehicle to mirror the physical robot's motion while the passenger simultaneously views the vehicle's first-person camera feed and its navigation decisions through an in-vehicle dashboard interface. We evaluate the framework over 20 repeated closed-loop navigation trials. The system achieves a mean state-update latency of 29.63 ms, a mean relative route-progress error of 2.28% between the physical and virtual vehicles, and video delivery at 10.006 frames per second with 0.25% frame loss. All monitored navigation decisions were correctly reflected in the VR interface with no missed or incorrect notifications. The results indicate that the framework can support temporally synchronized, semantically consistent, and accurate route-progress representation for immersive observation of physical autonomous-vehicle behavior.