mlsanityData to 7 Oct 2026

Reward-Relevance-Filtered Linear Offline Reinforcement Learning

arXiv 2401.12934 ↗First posted 23 Jan 2024stat.ML cs.LG math.OCcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

This paper studies offline reinforcement learning with linear function approximation in a setting with decision-theoretic, but not estimation sparsity. The structural restrictions of the data-generating process presume that the transitions factor into a sparse component that affects the reward and could affect additional exogenous dynamics that do not affect the reward. Although the minimally sufficient adjustment set for estimation of full-state transition properties depends on the whole state, the optimal policy and therefore state-action value function depends only on the sparse component: we call this causal/decision-theoretic sparsity. We develop a method for reward-filtering the estimation of the state-action value function to the sparse component by a modification of thresholded lasso in least-squares policy evaluation. We provide theoretical guarantees for our reward-filtered linear fitted-Q-iteration, with sample complexity depending only on the size of the sparse component.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 32 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

14 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning3 Nov
  2. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning25 Mar
  3. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient8 Nov

Other references · 11

  1. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  2. 2019Batch Policy Learning under Constraintscited by 242
  3. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  4. 2020Invariant Causal Prediction for Block MDPscited by 104
  5. 2021Causal Influence Detection for Improving Efficiency in Reinforcement Learningcited by 70
  6. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 66
  7. 2022Causal Dynamics Learning for Task-Independent State Abstractioncited by 64
  8. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 42
  9. 2018Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learningcited by 24
  10. 2020Thresholded Lasso Banditcited by 16
  11. 2021Sparsity in Partially Controllable Linear Systemscited by 9

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learningcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
  2. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity
  3. 2025What Fundamental Structure in Reward Functions Enables Efficient Sparse-Reward Learning?
  4. 2025Sparse Offline Reinforcement Learning with Corruption Robustness
  5. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning
  6. 2025Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings
  7. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
  8. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism