mlsanityData to 7 Oct 2026

Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning

arXiv 2111.08066 ↗First posted 15 Nov 2021cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning -- learning a policy from a batch of data -- is known to be hard for general MDPs. These results motivate the need to look at specific classes of MDPs where offline reinforcement learning might be feasible. In this work, we explore a restricted class of MDPs to obtain guarantees for offline reinforcement learning. The key property, which we call Action Impact Regularity (AIR), is that actions primarily impact a part of the state (an endogenous component) and have limited impact on the remaining part of the state (an exogenous component). AIR is a strong assumption, but it nonetheless holds in a number of real-world domains including financial markets. We discuss algorithms that exploit the AIR property, and provide a theoretical analysis for an algorithm based on Fitted-Q Iteration. Finally, we demonstrate that the algorithm outperforms existing offline reinforcement learning algorithms across different data collection policies in simulated and real world environments where the regularity holds.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 54 references that are in the map. First posted on arXiv 15 Nov 2021.

Builds on

23 of 54 references.

Closest ancestors · 10

Same direction, newest first

  1. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning26 Oct
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning25 Mar
  4. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  5. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 13

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  5. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  6. 2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 124
  7. 2019Learning with Good Feature Representations in Bandits and in RL with a Generative Modelcited by 124
  8. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  9. 2021Provable RL with Exogenous Distractors via Multistep Inverse Dynamicscited by 28
  10. 2018Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learningcited by 24
  11. 2020Efficient Planning in Large MDPs with Weak Linear Function Approximationcited by 17
  12. 2019Learning Compact Models for Planning with Exogenous Processescited by 11
1 more
  1. 2017Factoring Exogenous State for Model-Free Monte Carlocited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 0; 2026: 1.

2 papers.

  1. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  2. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces
  2. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  3. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  4. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
  5. 2021A Dataset Perspective on Offline Reinforcement Learning
  6. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  7. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
  8. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning