# Parallel and Multi-Objective Falsification with Scenic and VerifAI

arXiv: https://arxiv.org/abs/2107.04164 · HTML: https://mlsanity.com/paper/2107.04164/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 12 of 12)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [Scenic: A Language for Scenario Specification and Scene Generation](https://mlsanity.com/paper/1809.09310/) (2018)
- [Liability, Ethics, and Culture-Aware Behavior Specification using Rulebooks](https://mlsanity.com/paper/1902.09355/) (2019)
- [Scenic: A Language for Scenario Specification and Data Generation](https://mlsanity.com/paper/2010.06580/) (2020)
- [Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World](https://mlsanity.com/paper/2003.07739/) (2020)
- [Towards Verified Artificial Intelligence](https://mlsanity.com/paper/1606.08514/) (2016)
- [Securing Connected & Autonomous Vehicles: Challenges Posed by Adversarial Machine Learning and The Way Forward](https://mlsanity.com/paper/1905.12762/) (2019)
- [Upper-Confidence-Bound Algorithms for Active Learning in Multi-Armed Bandits](https://mlsanity.com/paper/1507.04523/) (2015)
- [Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAI](https://mlsanity.com/paper/2005.07173/) (2020)
- [Incremental Sampling-based Algorithm for Minimum-violation Motion Planning](https://mlsanity.com/paper/1305.1102/) (2013)

## Built on by (most cited first, 8 of 8)

- [Hierarchical Potential-based Reward Shaping from Task Specifications](https://mlsanity.com/paper/2110.02792/) (2021)
- [A Scenario-Based Platform for Testing Autonomous Vehicle Behavior Prediction Models in Simulation](https://mlsanity.com/paper/2110.14870/) (2021)
- [Addressing the IEEE AV Test Challenge with Scenic and VerifAI](https://mlsanity.com/paper/2108.13796/) (2021)
- [Risk-Aware Scene Sampling for Dynamic Assurance of Autonomous Systems](https://mlsanity.com/paper/2202.13510/) (2022)
- [Systems Challenges for Trustworthy Embodied Systems](https://mlsanity.com/paper/2201.03413/) (2022)
- [ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios](https://mlsanity.com/paper/2602.16073/) (2026)
- [Safe Policy Improvement in Constrained Markov Decision Processes](https://mlsanity.com/paper/2210.11259/) (2022)
- [Compositional Inductive Invariant Based Verification of Neural Network Controlled Systems](https://mlsanity.com/paper/2312.10842/) (2023)

## Abstract (arXiv)

Falsification has emerged as an important tool for simulation-based verification of autonomous systems. In this paper, we present extensions to the Scenic scenario specification language and VerifAI toolkit that improve the scalability of sampling-based falsification methods by using parallelism and extend falsification to multi-objective specifications. We first present a parallelized framework that is interfaced with both the simulation and sampling capabilities of Scenic and the falsification capabilities of VerifAI, reducing the execution time bottleneck inherently present in simulation-based testing. We then present an extension of VerifAI's falsification algorithms to support multi-objective optimization during sampling, using the concept of rulebooks to specify a preference ordering over multiple metrics that can be used to guide the counterexample search process. Lastly, we evaluate the benefits of these extensions with a comprehensive set of benchmarks written in the Scenic language.