# Scientific Discovery and the Cost of Measurement -- Balancing Information and Cost in Reinforcement Learning

arXiv: https://arxiv.org/abs/2112.07535 · HTML: https://mlsanity.com/paper/2112.07535/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 14 Dec 2021.

## Builds on (most cited first, 9 of 9)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [On Improving Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1804.06309/) (2018)
- [Active Reinforcement Learning: Observing Rewards at a Cost](https://mlsanity.com/paper/2011.06709/) (2020)
- [Active Measure Reinforcement Learning for Observation Cost Minimization](https://mlsanity.com/paper/2005.12697/) (2020)
- [Controlled Online Optimization Learning (COOL): Finding the ground state of spin Hamiltonians with reinforcement learning](https://mlsanity.com/paper/2003.00011/) (2020)
- [Active Reinforcement Learning with Monte-Carlo Tree Search](https://mlsanity.com/paper/1803.04926/) (2018)

## Built on by (most cited first, 4 of 4)

- [Dynamic Observation Policies in Observation Cost-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2307.02620/) (2023)
- [OCMDP: Observation-Constrained Markov Decision Process](https://mlsanity.com/paper/2411.07087/) (2024)
- [Active Measuring in Reinforcement Learning With Delayed Negative Effects](https://mlsanity.com/paper/2510.14315/) (2025)
- [Optimizing Algorithms for Mobile Health Interventions with Active Querying Optimization](https://mlsanity.com/paper/2512.08950/) (2025)

## Abstract (arXiv)

The use of reinforcement learning (RL) in scientific applications, such as materials design and automated chemistry, is increasing. A major challenge, however, lies in fact that measuring the state of the system is often costly and time consuming in scientific applications, whereas policy learning with RL requires a measurement after each time step. In this work, we make the measurement costs explicit in the form of a costed reward and propose a framework that enables off-the-shelf deep RL algorithms to learn a policy for both selecting actions and determining whether or not to measure the current state of the system at each time step. In this way, the agents learn to balance the need for information with the cost of information. Our results show that when trained under this regime, the Dueling DQN and PPO agents can learn optimal action policies whilst making up to 50% fewer state measurements, and recurrent neural networks can produce a greater than 50% reduction in measurements. We postulate the these reduction can help to lower the barrier to applying RL to real-world scientific applications.