# Continuous-Time Markov Decision Processes with Controlled Observations

arXiv: https://arxiv.org/abs/1907.06128 · HTML: https://mlsanity.com/paper/1907.06128/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2019.

## Built on by (most cited first, 7 of 7)

- [TempoRL: Learning When to Act](https://mlsanity.com/paper/2106.05262/) (2021)
- [Reinforcement Learning for Feedback-Enabled Cyber Resilience](https://mlsanity.com/paper/2107.00783/) (2021)
- [When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL](https://mlsanity.com/paper/2406.01163/) (2024)
- [Symbiotic Game and Foundation Models for Cyber Deception Operations in Strategic Cyber Warfare](https://mlsanity.com/paper/2403.10570/) (2024)
- [Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Act](https://mlsanity.com/paper/2203.08542/) (2022)
- [Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off](https://mlsanity.com/paper/2212.08949/) (2022)
- [Self-Triggered Markov Decision Processes](https://mlsanity.com/paper/2102.08571/) (2021)

## Abstract (arXiv)

In this paper, we study a continuous-time discounted jump Markov decision process with both controlled actions and observations. The observation is only available for a discrete set of time instances. At each time of observation, one has to select an optimal timing for the next observation and a control trajectory for the time interval between two observation points. We provide a theoretical framework that the decision maker can utilize to find the optimal observation epochs and the optimal actions jointly. Two cases are investigated. One is gated queueing systems in which we explicitly characterize the optimal action and the optimal observation where the optimal observation is shown to be independent of the state. Another is the inventory control problem with Poisson arrival process in which we obtain numerically the optimal action and observation. The results show that it is optimal to observe more frequently at a region of states where the optimal action adapts constantly.