mlsanityData to 14 Oct 2026

Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learning

arXiv 1806.01584 ↗First posted 5 Jun 2018cs.LG stat.MLcited by 24

Not on the map. Why?

Abstract (arXiv)

Exogenous state variables and rewards can slow down reinforcement learning by injecting uncontrolled variation into the reward signal. We formalize exogenous state variables and rewards and identify conditions under which an MDP with exogenous state can be decomposed into an exogenous Markov Reward Process involving only the exogenous state+reward and an endogenous Markov Decision Process defined with respect to only the endogenous rewards. We also derive a variance-covariance condition under which Monte Carlo policy evaluation on the endogenous MDP is accelerated compared to using the full MDP. Similar speedups are likely to carry over to all RL algorithms. We develop two algorithms for discovering the exogenous variables and test them on several MDPs. Results show that the algorithms are practical and can significantly speed up reinforcement learning.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 8; 2023: 7; 2024: 4; 2025: 0; 2026: 2.

24 papers.

Sort:

By area:

  1. 2021Learning to drive from a world on railscited by 114
  2. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 45
  3. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28
  4. 2022Sample-Efficient Reinforcement Learning in the Presence of Exogenous Informationcited by 26
  5. 2022Hindsight Learning for MDPs with Exogenous Inputscited by 26
  6. 2023Explainable Reinforcement Learning via a Causal World Modelcited by 22
  7. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  8. 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 16
  9. 2023Ignorance is Bliss: Robust Control via Information Gatingcited by 10
  10. 2023A Versatile Multi-Agent Reinforcement Learning Benchmark for Inventory Managementcited by 10
  11. 2024Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learningcited by 9
  12. 2023Online Reinforcement Learning in Non-Stationary Context-Driven Environmentscited by 6
12 more
  1. 2022A Reinforcement Learning Approach to Estimating Long-term Treatment Effectscited by 5
  2. 2023Towards Generalizable Reinforcement Learning for Trade Executioncited by 5
  3. 2023Reinforcement Learning with Exogenous States and Rewardscited by 4
  4. 2024Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learningcited by 3
  5. 2021Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learningcited by 2
  6. 2024Generalizing Multi-Step Inverse Models for Representation Learning to Finite-Memory POMDPscited by 2
  7. 2022A Bayesian Approach to Learning Bandit Structure in Markov Decision Processescited by 1
  8. 2023Reinforcement Learning under External Influence: Guarantees, Algorithms, and Sample Complexitycited by 1
  9. 2024Reward-Relevance-Filtered Linear Offline Reinforcement Learningcited by 1
  10. 2026Graphon Mean-Field Subsampling for Cooperative Heterogeneous Multi-Agent Reinforcement Learningcited by 1
  11. 2022The Sandbox Environment for Generalizable Agent Research (SEGAR)cited by 0
  12. 2026Is Pure Exploitation Sufficient in Exogenous MDPs with Linear Function Approximation?cited by 0