# Configurable Markov Decision Processes

arXiv: https://arxiv.org/abs/1806.05415 · HTML: https://mlsanity.com/paper/1806.05415/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2018.

## Built on by (most cited first, 9 of 9)

- [Unsupervised Reinforcement Learning in Multiple Environments](https://mlsanity.com/paper/2112.08746/) (2021)
- [Data Duplication: A Novel Multi-Purpose Attack Paradigm in Machine Unlearning](https://mlsanity.com/paper/2501.16663/) (2025)
- [Environment Design for Inverse Reinforcement Learning](https://mlsanity.com/paper/2210.14972/) (2022)
- [Simultaneously Updating All Persistence Values in Reinforcement Learning](https://mlsanity.com/paper/2211.11620/) (2022)
- [Model-Advantage and Value-Aware Models for Model-Based Reinforcement Learning: Bridging the Gap in Theory and Practice](https://mlsanity.com/paper/2106.14080/) (2021)
- [Tracking Control for a Spherical Pendulum via Curriculum Reinforcement Learning](https://mlsanity.com/paper/2309.14096/) (2023)
- [Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions](https://mlsanity.com/paper/2510.15056/) (2025)
- [Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning](https://mlsanity.com/paper/2107.05798/) (2021)
- [Optimizing Empty Container Repositioning and Fleet Deployment via Configurable Semi-POMDPs](https://mlsanity.com/paper/2207.12509/) (2022)

## Abstract (arXiv)

In many real-world problems, there is the possibility to configure, to a limited extent, some environmental parameters to improve the performance of a learning agent. In this paper, we propose a novel framework, Configurable Markov Decision Processes (Conf-MDPs), to model this new type of interaction with the environment. Furthermore, we provide a new learning algorithm, Safe Policy-Model Iteration (SPMI), to jointly and adaptively optimize the policy and the environment configuration. After having introduced our approach and derived some theoretical results, we present the experimental evaluation in two explicative problems to show the benefits of the environment configurability on the performance of the learned policy.