# Learning Compact Models for Planning with Exogenous Processes

arXiv: https://arxiv.org/abs/1909.13870 · HTML: https://mlsanity.com/paper/1909.13870/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2019.

## Built on by (most cited first, 11 of 11)

- [Learning to drive from a world on rails](https://mlsanity.com/paper/2105.00636/) (2021)
- [Hindsight Learning for MDPs with Exogenous Inputs](https://mlsanity.com/paper/2207.06272/) (2022)
- [Learning Latent Dynamic Robust Representations for World Models](https://mlsanity.com/paper/2405.06263/) (2024)
- [A Review of Reinforcement Learning in Financial Applications](https://mlsanity.com/paper/2411.12746/) (2024)
- [Deep Reinforcement Learning Aided Platoon Control Relying on V2X Information](https://mlsanity.com/paper/2203.15781/) (2022)
- [A Reinforcement Learning Approach to Estimating Long-term Treatment Effects](https://mlsanity.com/paper/2210.07536/) (2022)
- [Reinforcement Learning with Exogenous States and Rewards](https://mlsanity.com/paper/2303.12957/) (2023)
- [Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning](https://mlsanity.com/paper/2111.08066/) (2021)
- [Influence-Augmented Local Simulators: A Scalable Solution for Fast Deep RL in Large Networked Systems](https://mlsanity.com/paper/2202.01534/) (2022)
- [A Bayesian Approach to Learning Bandit Structure in Markov Decision Processes](https://mlsanity.com/paper/2208.00250/) (2022)
- [Online Planning in POMDPs with Self-Improving Simulators](https://mlsanity.com/paper/2201.11404/) (2022)

## Abstract (arXiv)

We address the problem of approximate model minimization for MDPs in which the state is partitioned into endogenous and (much larger) exogenous components. An exogenous state variable is one whose dynamics are independent of the agent's actions. We formalize the mask-learning problem, in which the agent must choose a subset of exogenous state variables to reason about when planning; doing planning in such a reduced state space can often be significantly more efficient than planning in the full model. We then explore the various value functions at play within this setting, and describe conditions under which a policy for a reduced model will be optimal for the full MDP. The analysis leads us to a tractable approximate algorithm that draws upon the notion of mutual information among exogenous state variables. We validate our approach in simulated robotic manipulation domains where a robot is placed in a busy environment, in which there are many other agents also interacting with the objects. Visit http://tinyurl.com/chitnis-exogenous for a supplementary video.