# Active Measure Reinforcement Learning for Observation Cost Minimization

arXiv: https://arxiv.org/abs/2005.12697 · HTML: https://mlsanity.com/paper/2005.12697/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2020.

## Built on by (most cited first, 17 of 17)

- [Automated Scientific Discovery: From Equation Discovery to Autonomous Discovery Systems](https://mlsanity.com/paper/2305.02251/) (2023)
- [Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuring](https://mlsanity.com/paper/2303.08271/) (2023)
- [Monitored Markov Decision Processes](https://mlsanity.com/paper/2402.06819/) (2024)
- [Scientific Discovery and the Cost of Measurement -- Balancing Information and Cost in Reinforcement Learning](https://mlsanity.com/paper/2112.07535/) (2021)
- [Dynamic Observation Policies in Observation Cost-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2307.02620/) (2023)
- [Towards Cost Sensitive Decision Making](https://mlsanity.com/paper/2410.03892/) (2024)
- [OCMDP: Observation-Constrained Markov Decision Process](https://mlsanity.com/paper/2411.07087/) (2024)
- [Revelations: A Decidable Class of POMDPs with Omega-Regular Objectives](https://mlsanity.com/paper/2412.12063/) (2024)
- [Attention when you need](https://mlsanity.com/paper/2501.07440/) (2025)
- [Dynamic programming with incomplete information to overcome navigational uncertainty in a nautical environment](https://mlsanity.com/paper/2112.14657/) (2021)
- [Robust Active Measuring under Model Uncertainty](https://mlsanity.com/paper/2312.11227/) (2023)
- [Online Planning in POMDPs with State-Requests](https://mlsanity.com/paper/2407.18812/) (2024)
- [MDPs with a State Sensing Cost](https://mlsanity.com/paper/2505.03280/) (2025)
- [Reinforcement Learning with Action-Triggered Observations](https://mlsanity.com/paper/2510.02149/) (2025)
- [Active Measuring in Reinforcement Learning With Delayed Negative Effects](https://mlsanity.com/paper/2510.14315/) (2025)
- [Optimizing Algorithms for Mobile Health Interventions with Active Querying Optimization](https://mlsanity.com/paper/2512.08950/) (2025)
- [On Training in Imagination](https://mlsanity.com/paper/2605.06732/) (2026)

## Abstract (arXiv)

Standard reinforcement learning (RL) algorithms assume that the observation of the next state comes instantaneously and at no cost. In a wide variety of sequential decision making tasks ranging from medical treatment to scientific discovery, however, multiple classes of state observations are possible, each of which has an associated cost. We propose the active measure RL framework (Amrl) as an initial solution to this problem where the agent learns to maximize the costed return, which we define as the discounted sum of rewards minus the sum of observation costs. Our empirical evaluation demonstrates that Amrl-Q agents are able to learn a policy and state estimator in parallel during online training. During training the agent naturally shifts from its reliance on costly measurements of the environment to its state estimator in order to increase its reward. It does this without harm to the learned policy. Our results show that the Amrl-Q agent learns at a rate similar to standard Q-learning and Dyna-Q. Critically, by utilizing an active strategy, Amrl-Q achieves a higher costed return.