# Sim-on-Wheels: Physical World in the Loop Simulation for Self-Driving

arXiv: https://arxiv.org/abs/2306.08807 · HTML: https://mlsanity.com/paper/2306.08807/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2023.

## Built on by (most cited first, 7 of 7)

- [LidarDM: Generative LiDAR Simulation in a Generated World](https://mlsanity.com/paper/2404.02903/) (2024)
- [Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation](https://mlsanity.com/paper/2501.06693/) (2025)
- [AutoVFX: Physically Realistic Video Editing from Natural Language Instructions](https://mlsanity.com/paper/2411.02394/) (2024)
- [SuperGaussian: Repurposing Video Models for 3D Super Resolution](https://mlsanity.com/paper/2406.00609/) (2024)
- [Navigating the Edge with the State-of-the-Art Insights into Corner Case Identification and Generation for Enhanced Autonomous Vehicle Safety](https://mlsanity.com/paper/2503.00077/) (2025)
- [A Multi-Modality Evaluation of the Reality Gap in Autonomous Driving Systems](https://mlsanity.com/paper/2509.22379/) (2025)
- [SimPRIVE: a Simulation framework for Physical Robot Interaction with Virtual Environments](https://mlsanity.com/paper/2504.21454/) (2025)

## Abstract (arXiv)

We present Sim-on-Wheels, a safe, realistic, and vehicle-in-loop framework to test autonomous vehicles' performance in the real world under safety-critical scenarios. Sim-on-wheels runs on a self-driving vehicle operating in the physical world. It creates virtual traffic participants with risky behaviors and seamlessly inserts the virtual events into images perceived from the physical world in real-time. The manipulated images are fed into autonomy, allowing the self-driving vehicle to react to such virtual events. The full pipeline runs on the actual vehicle and interacts with the physical world, but the safety-critical events it sees are virtual. Sim-on-Wheels is safe, interactive, realistic, and easy to use. The experiments demonstrate the potential of Sim-on-Wheels to facilitate the process of testing autonomous driving in challenging real-world scenes with high fidelity and low risk.