mlsanityData to 7 Oct 2026

CounterScene: Counterfactual Causal Reasoning in Generative World Models for Safety-Critical Closed-Loop Evaluation

arXiv 2603.21104 ↗First posted 22 Mar 2026cs.RO cs.CVcited by 3

On the map: World models → Latent predictive world models

also in 1 more
Abstract (arXiv)

Generating safety-critical driving scenarios requires understanding why dangerous interactions arise, rather than merely forcing collisions. However, existing methods rely on heuristic adversarial agent selection and unstructured perturbations, lacking explicit modeling of interaction dependencies and thus exhibiting a realism--adversarial trade-off. We present CounterScene, a framework that endows closed-loop generative BEV world models with structured counterfactual reasoning for safety-critical scenario generation. Given a safe scene, CounterScene asks: what if the causally critical agent had behaved differently? To answer this, we introduce causal adversarial agent identification to identify the critical agent and classify conflict types, and develop a conflict-aware interactive world model in which a causal interaction graph is used to explicitly model dynamic inter-agent dependencies. Building on this structure, stage-adaptive counterfactual guidance performs minimal interventions on the identified agent, removing its spatial and temporal safety margins while allowing risk to emerge through natural interaction propagation. Extensive experiments on nuScenes demonstrate that CounterScene achieves the strongest adversarial effectiveness while maintaining superior trajectory realism across all horizons, improving long-horizon collision rate from 12.3% to 22.7% over the strongest baseline with better realism (ADE 1.88 vs.2.09). Notably, this advantage further widens over longer rollouts, and CounterScene generalizes zero-shot to nuPlan with state-of-the-art realism.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 41 of its 58 references that are in the map. First posted on arXiv 22 Mar 2026.

Builds on

41 of 58 references.

References · 41

  1. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  4. 2021NuPlan: A closed-loop ML-based planning benchmark for autonomous vehiclescited by 485
  5. 2024NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarkingcited by 343
  6. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  7. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  8. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  9. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 230
  10. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
  11. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  12. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 178
29 more
  1. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 165
  2. 2025End-to-End Driving with Online Trajectory Evaluation via BEV World Modelcited by 145
  3. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
  4. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  5. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
  6. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 110
  7. 2022A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspectivecited by 100
  8. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 81
  9. 2022M2I: From Factored Marginal Trajectory Prediction to Interactive Predictioncited by 81
  10. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  11. 2024GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Predictioncited by 66
  12. 2022SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehiclescited by 65
  13. 2024NeuroNCAP: Photorealistic Closed-loop Safety Testing for Autonomous Drivingcited by 61
  14. 2025HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generationcited by 56
  15. 2016Accelerated Evaluation of Automated Vehicles Safety in Lane Change Scenarios Based on Importance Sampling Techniquescited by 44
  16. 2023SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversariescited by 36
  17. 2025MaskGWM: A Generalizable Driving World Model with Video Mask Reconstructioncited by 27
  18. 2024Causal Composition Diffusion Model for Closed-loop Traffic Generationcited by 22
  19. 2023Learning Cooperative Trajectory Representations for Motion Forecastingcited by 21
  20. 2021CausalAF: Causal Autoregressive Flow for Safety-Critical Driving Scenario Generationcited by 20
  21. 2025StyleDrive: Towards Driving-Style Aware Benchmarking of End-To-End Autonomous Drivingcited by 20
  22. 2025Trajectory Mamba: Efficient Attention-Mamba Forecasting Model Based on Selective SSMcited by 17
  23. 2024Adversarial Safety-Critical Scenario Generation using Naturalistic Human Driving Priorscited by 15
  24. 2025CurricuVLM: Towards Safe Autonomous Driving via Personalized Safety-Critical Curriculum Learning with Vision-Language Modelscited by 11
  25. 2025DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulationcited by 5
  26. 2026ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstructioncited by 5
  27. 2024Barrier-Enhanced Parallel Homotopic Trajectory Optimization for Safety-Critical Autonomous Drivingcited by 4
  28. 2023A Counterfactual Safety Margin Perspective on the Scoring of Autonomous Vehicles' Riskinesscited by 3
  29. 2025HAD-Gen: Human-like and Diverse Driving Behavior Modeling for Controllable Scenario Generationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Adversarial Generation and Collaborative Evolution of Safety-Critical Scenarios for Autonomous Vehicles
  2. 2026Safety-Critical Scenarios Emerge from Initial Scenes ↗
  3. 2025Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviour
  4. 2026Low-Rank Dynamics-Effective Latent Carriers for Counterfactual Rollout in Learned World Models
  5. 2026Rethinking World Models for Safety-Critical Embodied Systems
  6. 2026Learning Counterfactual World Models for Embodied Reasoning under Partial Observability
  7. 2026Should I Have Expressed a Different Intent? Counterfactual Generation for LLM-Based Autonomous Control
  8. 2026C-CoT: Counterfactual Chain-of-Thought with Vision-Language Models for Safe Autonomous Driving