# POMDPs in Continuous Time and Discrete Spaces

arXiv: https://arxiv.org/abs/2010.01014 · HTML: https://mlsanity.com/paper/2010.01014/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 3 of 3)

- [Reinforcement Learning with Non-Exponential Discounting](https://mlsanity.com/paper/2209.13413/) (2022)
- [Engagement Process: Rethinking the Temporal Interface of Action and Observation](https://mlsanity.com/paper/2605.11484/) (2026)
- [Approximate Control for Continuous-Time POMDPs](https://mlsanity.com/paper/2402.01431/) (2024)

## Abstract (arXiv)

Many processes, such as discrete event systems in engineering or population dynamics in biology, evolve in discrete space and continuous time. We consider the problem of optimal decision making in such discrete state and action space systems under partial observability. This places our work at the intersection of optimal filtering and optimal control. At the current state of research, a mathematical description for simultaneous decision making and filtering in continuous time with finite state and action spaces is still missing. In this paper, we give a mathematical description of a continuous-time partial observable Markov decision process (POMDP). By leveraging optimal filtering theory we derive a Hamilton-Jacobi-Bellman (HJB) type equation that characterizes the optimal solution. Using techniques from deep learning we approximately solve the resulting partial integro-differential equation. We present (i) an approach solving the decision problem offline by learning an approximation of the value function and (ii) an online algorithm which provides a solution in belief space using deep reinforcement learning. We show the applicability on a set of toy examples which pave the way for future methods providing solutions for high dimensional problems.