# Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environments

arXiv: https://arxiv.org/abs/2106.10365 · HTML: https://mlsanity.com/paper/2106.10365/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 18 Jun 2021.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning](https://mlsanity.com/paper/1910.05396/) (2019)
- [Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2005.06800/) (2020)
- [Scenic: A Language for Scenario Specification and Scene Generation](https://mlsanity.com/paper/1809.09310/) (2018)
- [Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation](https://mlsanity.com/paper/1806.10729/) (2018)
- [Gotta Learn Fast: A New Benchmark for Generalization in RL](https://mlsanity.com/paper/1804.03720/) (2018)
- [Scenic: A Language for Scenario Specification and Data Generation](https://mlsanity.com/paper/2010.06580/) (2020)
- [Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning](https://mlsanity.com/paper/2010.13303/) (2020)
- [Natural Environment Benchmarks for Reinforcement Learning](https://mlsanity.com/paper/1811.06032/) (2018)
- [Paracosm: A Language and Tool for Testing Autonomous Driving Systems](https://mlsanity.com/paper/1902.01084/) (2019)

## Built on by (most cited first, 6 of 6)

- [Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search](https://mlsanity.com/paper/2405.15383/) (2024)
- [CLUTR: Curriculum Learning via Unsupervised Task Representation Learning](https://mlsanity.com/paper/2210.10243/) (2022)
- [ShuttleSet: A Human-Annotated Stroke-Level Singles Dataset for Badminton Tactical Analysis](https://mlsanity.com/paper/2306.04948/) (2023)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [MuLMINet: Multi-Layer Multi-Input Transformer Network with Weighted Loss](https://mlsanity.com/paper/2307.08262/) (2023)
- [SDGym: Low-Code Reinforcement Learning Environments using System Dynamics Models](https://mlsanity.com/paper/2310.12494/) (2023)

## Abstract (arXiv)

The capability of a reinforcement learning (RL) agent heavily depends on the diversity of the learning scenarios generated by the environment. Generation of diverse realistic scenarios is challenging for real-time strategy (RTS) environments. The RTS environments are characterized by intelligent entities/non-RL agents cooperating and competing with the RL agents with large state and action spaces over a long period of time, resulting in an infinite space of feasible, but not necessarily realistic, scenarios involving complex interaction among different RL and non-RL agents. Yet, most of the existing simulators rely on randomly generating the environments based on predefined settings/layouts and offer limited flexibility and control over the environment dynamics for researchers to generate diverse, realistic scenarios as per their demand. To address this issue, for the first time, we formally introduce the benefits of adopting an existing formal scenario specification language, SCENIC, to assist researchers to model and generate diverse scenarios in an RTS environment in a flexible, systematic, and programmatic manner. To showcase the benefits, we interfaced SCENIC to an existing RTS environment Google Research Football(GRF) simulator and introduced a benchmark consisting of 32 realistic scenarios, encoded in SCENIC, to train RL agents and testing their generalization capabilities. We also show how researchers/RL practitioners can incorporate their domain knowledge to expedite the training process by intuitively modeling stochastic programmatic policies with SCENIC.