# Video2Track: From Real-World Interaction Videos to Steerable Adversarial Closed-Track Testing for Automated Driving Systems

arXiv: https://arxiv.org/abs/2608.11592 · HTML: https://mlsanity.com/paper/2608.11592/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Adversarial scenarios and safety testing, one of 9 directions in End-to-end autonomous driving.
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Placed in

- End-to-end autonomous driving / [Adversarial scenarios and safety testing](https://mlsanity.com/driving/adversarial-scenarios-and-safety-testing/)

## Builds on (most cited first, 11 of 11)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios](https://mlsanity.com/paper/2210.06609/) (2022)
- [A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective](https://mlsanity.com/paper/2202.02215/) (2022)
- [Scenic: A Language for Scenario Specification and Scene Generation](https://mlsanity.com/paper/1809.09310/) (2018)
- ["Curse of rarity" for autonomous vehicles](https://mlsanity.com/paper/2207.02749/) (2022)
- [Game-Theoretic Modeling of Driver and Vehicle Interactions for Verification and Validation of Autonomous Vehicle Control Systems](https://mlsanity.com/paper/1608.08589/) (2016)
- [Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Driving](https://mlsanity.com/paper/2501.08096/) (2025)
- [DriveGen: Towards Infinite Diverse Traffic Scenarios with Large Models](https://mlsanity.com/paper/2503.05808/) (2025)
- [Real-world Troublemaker: A 5G Cloud-controlled Track Testing Framework for Automated Driving Systems in Safety-critical Interaction Scenarios](https://mlsanity.com/paper/2502.14574/) (2025)
- [An Evolving Scenario Generation Method based on Dual-modal Driver Model Trained by Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.02027/) (2025)

## Built on by (most cited first, 1 of 1)

- [SSP: An Event-Matched Syn2Sim2Phy Cross-Domain Evaluation Framework for Autonomous Driving VLA Models](https://mlsanity.com/paper/2608.14024/) (2026)

## Abstract (arXiv)

Closed-track testing plays a fundamental role in the verification and validation of automated driving systems (ADS), particularly for safety-critical scenarios, by enabling reproducible evaluation under controlled conditions. However, most existing approaches still rely on standardized protocols or predefined trajectories, leading to overly scripted interactions and limited ability to reproduce the natural complexity of public-road traffic. To address this limitation, we propose Video2Track, a framework that transfers real-world interactive driving scenarios from videos into steerable adversarial closed-track testing. The framework consists of two tightly coupled modules. The first is a scenario semantic mapping module, which extracts structured semantics from driving videos using a vision-language model and grounds them onto a closed-track topology library via retrieval-augmented generation, thereby identifying compatible map segments and interaction anchors. The second is a dynamic interactive testing module, which conditions on the grounded topology and anchors to generate diverse multi-agent trajectories through a conditional diffusion model, while regulating interaction intensity via a Stackelberg game with a parameterized adversarial objective. Closed-track experiments demonstrate that the proposed framework can faithfully reproduce representative real-world interaction scenarios and generate executable scenario variants with controllable risk levels and interaction styles, providing a scalable approach for realistic and steerable ADS validation.