# Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions

arXiv: https://arxiv.org/abs/2510.15056 · HTML: https://mlsanity.com/paper/2510.15056/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 2 of its 16 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 2 of 2)

- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Configurable Markov Decision Processes](https://mlsanity.com/paper/1806.05415/) (2018)

## Built on by (most cited first, 1 of 1)

- [AGWM: Affordance-Grounded World Models for Environments with Compositional Prerequisites](https://mlsanity.com/paper/2605.06841/) (2026)

## Abstract (arXiv)

Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.