# OCMDP: Observation-Constrained Markov Decision Process

arXiv: https://arxiv.org/abs/2411.07087 · HTML: https://mlsanity.com/paper/2411.07087/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 11 Nov 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Reinforcement learning](https://mlsanity.com/paper/2405.10369/) (2024)
- [DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents](https://mlsanity.com/paper/2410.14803/) (2024)
- [An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2011.11235/) (2020)
- [Reinforcement Learning with Efficient Active Feature Acquisition](https://mlsanity.com/paper/2011.00825/) (2020)
- [Active Measure Reinforcement Learning for Observation Cost Minimization](https://mlsanity.com/paper/2005.12697/) (2020)
- [ASAC: Active Sensing using Actor-Critic models](https://mlsanity.com/paper/1906.06796/) (2019)
- [The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulation](https://mlsanity.com/paper/2106.04240/) (2021)
- [Challenges for Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2103.05612/) (2021)
- [Learning for Robot Decision Making under Distribution Shift: A Survey](https://mlsanity.com/paper/2203.07558/) (2022)
- [Scientific Discovery and the Cost of Measurement -- Balancing Information and Cost in Reinforcement Learning](https://mlsanity.com/paper/2112.07535/) (2021)
- [Learned Graph Rewriting with Equality Saturation: A New Paradigm in Relational Query Rewrite and Beyond](https://mlsanity.com/paper/2407.12794/) (2024)
- [IA2: Leveraging Instance-Aware Index Advisor with Reinforcement Learning for Diverse Workloads](https://mlsanity.com/paper/2404.05777/) (2024)

## Built on by (most cited first, 2 of 2)

- [Reinforcement Learning with Action-Triggered Observations](https://mlsanity.com/paper/2510.02149/) (2025)
- [Robust Remote Reinforcement Learning over Unreliable Communication Channels using Homomorphic State Encoding](https://mlsanity.com/paper/2508.07722/) (2025)

## Abstract (arXiv)

In many practical applications, decision-making processes must balance the costs of acquiring information with the benefits it provides. Traditional control systems often assume full observability, an unrealistic assumption when observations are expensive. We tackle the challenge of simultaneously learning observation and control strategies in such cost-sensitive environments by introducing the Observation-Constrained Markov Decision Process (OCMDP), where the policy influences the observability of the true state. To manage the complexity arising from the combined observation and control actions, we develop an iterative, model-free deep reinforcement learning algorithm that separates the sensing and control components of the policy. This decomposition enables efficient learning in the expanded action space by focusing on when and what to observe, as well as determining optimal control actions, without requiring knowledge of the environment's dynamics. We validate our approach on a simulated diagnostic task and a realistic healthcare environment using HeartPole. Given both scenarios, the experimental results demonstrate that our model achieves a substantial reduction in observation costs on average, significantly outperforming baseline methods by a notable margin in efficiency.