# Active Measuring in Reinforcement Learning With Delayed Negative Effects

arXiv: https://arxiv.org/abs/2510.14315 · HTML: https://mlsanity.com/paper/2510.14315/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 21 of 21)

- [Generalization and Exploration via Randomized Value Functions](https://mlsanity.com/paper/1402.0635/) (2014)
- [When Is Partially Observable Reinforcement Learning Not Scary?](https://mlsanity.com/paper/2204.08967/) (2022)
- [Active Preference Optimization for Sample Efficient RLHF](https://mlsanity.com/paper/2402.10500/) (2024)
- [On Particle Methods for Parameter Estimation in State-Space Models](https://mlsanity.com/paper/1412.8695/) (2014)
- [Reinforcement Learning from Human Feedback with Active Queries](https://mlsanity.com/paper/2402.09401/) (2024)
- [Information Directed Reward Learning for Reinforcement Learning](https://mlsanity.com/paper/2102.12466/) (2021)
- [Targeting for long-term outcomes](https://mlsanity.com/paper/2010.15835/) (2020)
- [Active Reinforcement Learning: Observing Rewards at a Cost](https://mlsanity.com/paper/2011.06709/) (2020)
- [Optimality Guarantees for Particle Belief Approximation of POMDPs](https://mlsanity.com/paper/2210.05015/) (2022)
- [Dual Active Learning for Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2410.02504/) (2024)
- [Active Measure Reinforcement Learning for Observation Cost Minimization](https://mlsanity.com/paper/2005.12697/) (2020)
- [Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning](https://mlsanity.com/paper/2304.08944/) (2023)
- [Active Reinforcement Learning with Monte-Carlo Tree Search](https://mlsanity.com/paper/1803.04926/) (2018)
- [Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuring](https://mlsanity.com/paper/2303.08271/) (2023)
- [Adaptive Informative Path Planning with Multimodal Sensing](https://mlsanity.com/paper/2003.09746/) (2020)
- [Beyond Optimism: Exploration With Partially Observable Rewards](https://mlsanity.com/paper/2406.13909/) (2024)
- [Periodic agent-state based Q-learning for POMDPs](https://mlsanity.com/paper/2407.06121/) (2024)
- [Harnessing Causality in Reinforcement Learning With Bagged Decision Times](https://mlsanity.com/paper/2410.14659/) (2024)
- [Scientific Discovery and the Cost of Measurement -- Balancing Information and Cost in Reinforcement Learning](https://mlsanity.com/paper/2112.07535/) (2021)
- [Robust Active Measuring under Model Uncertainty](https://mlsanity.com/paper/2312.11227/) (2023)
- [Online Planning in POMDPs with State-Requests](https://mlsanity.com/paper/2407.18812/) (2024)

## Built on by (most cited first, 1 of 1)

- [Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions](https://mlsanity.com/paper/2601.15353/) (2026)

## Abstract (arXiv)

Measuring states in reinforcement learning (RL) can be costly in real-world settings and may negatively influence future outcomes. We introduce the Actively Observable Markov Decision Process (AOMDP), where an agent not only selects control actions but also decides whether to measure the latent state. The measurement action reveals the true latent state but may have a negative delayed effect on the environment. We show that this reduced uncertainty may provably improve sample efficiency and increase the value of the optimal policy despite these costs. We formulate an AOMDP as a periodic partially observable MDP and propose an online RL algorithm based on belief states. To approximate the belief states, we further propose a sequential Monte Carlo method to jointly approximate the posterior of unknown static environment parameters and unobserved latent states. We evaluate the proposed algorithm in a digital health application, where the agent decides when to deliver digital interventions and when to assess users' health status through surveys.