mlsanityData to 14 Oct 2026

Active Measuring in Reinforcement Learning With Delayed Negative Effects

arXiv 2510.14315 ↗First posted 16 Oct 2025cs.LG stat.MLcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Measuring states in reinforcement learning (RL) can be costly in real-world settings and may negatively influence future outcomes. We introduce the Actively Observable Markov Decision Process (AOMDP), where an agent not only selects control actions but also decides whether to measure the latent state. The measurement action reveals the true latent state but may have a negative delayed effect on the environment. We show that this reduced uncertainty may provably improve sample efficiency and increase the value of the optimal policy despite these costs. We formulate an AOMDP as a periodic partially observable MDP and propose an online RL algorithm based on belief states. To approximate the belief states, we further propose a sequential Monte Carlo method to jointly approximate the posterior of unknown static environment parameters and unobserved latent states. We evaluate the proposed algorithm in a digital health application, where the agent decides when to deliver digital interventions and when to assess users' health status through surveys.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 36 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

21 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Harnessing Causality in Reinforcement Learning With Bagged Decision Times18 Oct
  2. 2024Periodic agent-state based Q-learning for POMDPs8 Jul

Other references · 19

  1. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  2. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  3. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  4. 2014On Particle Methods for Parameter Estimation in State-Space Modelscited by 47
  5. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 43
  6. 2021Information Directed Reward Learning for Reinforcement Learningcited by 24
  7. 2020Targeting for long-term outcomescited by 23
  8. 2020Active Reinforcement Learning: Observing Rewards at a Costcited by 20
  9. 2022Optimality Guarantees for Particle Belief Approximation of POMDPscited by 20
  10. 2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 20
  11. 2020Active Measure Reinforcement Learning for Observation Cost Minimizationcited by 17
  12. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
7 more
  1. 2018Active Reinforcement Learning with Monte-Carlo Tree Searchcited by 11
  2. 2023Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuringcited by 10
  3. 2020Adaptive Informative Path Planning with Multimodal Sensingcited by 9
  4. 2024Beyond Optimism: Exploration With Partially Observable Rewardscited by 6
  5. 2021Scientific Discovery and the Cost of Measurement -- Balancing Information and Cost in Reinforcement Learningcited by 4
  6. 2023Robust Active Measuring under Model Uncertaintycited by 1
  7. 2024Online Planning in POMDPs with State-Requestscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes
  2. 2025Statistical and Algorithmic Foundations of Reinforcement Learning
  3. 2024Monitored Markov Decision Processes
  4. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPs
  5. 2025Model-Based Exploration in Monitored Markov Decision Processes
  6. 2023Safe Policy Improvement for POMDPs via Finite-State Controllers
  7. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
  8. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees