mlsanityData to 14 Oct 2026

Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

arXiv 2607.25241 ↗First posted 28 Jul 2026stat.ML cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 4 of its 58 references that are in the map. First posted on arXiv 28 Jul 2026.

Builds on

4 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024A Tale of Two Cities: Pessimism and Opportunism in Offline Dynamic Pricing12 Nov
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 2

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  2. 2024Causal Inference with a Hidden Treatmentcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding
  2. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes
  3. 2020Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies
  4. 2025Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learning
  5. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
  6. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  7. 2023Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
  8. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption