# MDP environments for the OpenAI Gym

arXiv: https://arxiv.org/abs/1709.09069 · HTML: https://mlsanity.com/paper/1709.09069/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2017.

## Built on by (most cited first, 3 of 3)

- [Off-Policy Actor-Critic with Emphatic Weightings](https://mlsanity.com/paper/2111.08172/) (2021)
- [Curious Explorer: a provable exploration strategy in Policy Learning](https://mlsanity.com/paper/2106.15503/) (2021)
- [Optimising Call Centre Operations using Reinforcement Learning: Value Iteration versus Proximal Policy Optimisation](https://mlsanity.com/paper/2507.18398/) (2025)

## Abstract (arXiv)

The OpenAI Gym provides researchers and enthusiasts with simple to use environments for reinforcement learning. Even the simplest environment have a level of complexity that can obfuscate the inner workings of RL approaches and make debugging difficult. This whitepaper describes a Python framework that makes it very easy to create simple Markov-Decision-Process environments programmatically by specifying state transitions and rewards of deterministic and non-deterministic MDPs in a domain-specific language in Python. It then presents results and visualizations created with this MDP framework.