mlsanityData to 14 Oct 2026

Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions

arXiv 2510.15056 ↗First posted 16 Oct 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 2 of its 16 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

2 of 16 references.

References · 2

  1. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  2. 2018Configurable Markov Decision Processescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AGWM: Affordance-Grounded World Models for Environments with Compositional Prerequisitescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-level meta-reinforcement learning with skill-based curriculum
  2. 2026Adaptive Multi-Horizon Reinforcement Learning
  3. 2024PWM: Policy Learning with Multi-Task World Models
  4. 2025Model-Based Reinforcement Learning Under Confounding
  5. 2026Learning in Markov Decision Processes with Exogenous Dynamics
  6. 2025Reinforcement Learning with Imperfect Transition Predictions: A Bellman-Jensen Approach
  7. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control
  8. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization