# Partially Observed Markov Decision Processes. Problem Sets and Internet Supplement

arXiv: https://arxiv.org/abs/1604.08127 · HTML: https://mlsanity.com/paper/1604.08127/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2016.

## Built on by (most cited first, 30 of 34)

- [Convergence of Finite Memory Q-Learning for POMDPs and Near Optimality of Learned Policies under Filter Stability](https://mlsanity.com/paper/2103.12158/) (2021)
- [Intrusion Prevention through Optimal Stopping](https://mlsanity.com/paper/2111.00289/) (2021)
- [Learning Near-Optimal Intrusion Responses Against Dynamic Attackers](https://mlsanity.com/paper/2301.06085/) (2023)
- [Conjectural Online Learning with First-order Beliefs in Asymmetric Information Stochastic Games](https://mlsanity.com/paper/2402.18781/) (2024)
- [Identifying Cognitive Radars -- Inverse Reinforcement Learning using Revealed Preferences](https://mlsanity.com/paper/1912.00331/) (2019)
- [Automated Security Response through Online Learning with Adaptive Conjectures](https://mlsanity.com/paper/2402.12499/) (2024)
- [Optimal Defender Strategies for CAGE-2 using Causal Modeling and Tree Search](https://mlsanity.com/paper/2407.11070/) (2024)
- [Learning Intrusion Prevention Policies through Optimal Stopping](https://mlsanity.com/paper/2106.07160/) (2021)
- [Scalable Learning of Intrusion Responses through Recursive Decomposition](https://mlsanity.com/paper/2309.03292/) (2023)
- [Intrusion Tolerance for Networked Systems through Two-Level Feedback Control](https://mlsanity.com/paper/2404.01741/) (2024)
- [Learning Security Strategies through Game Play and Optimal Stopping](https://mlsanity.com/paper/2205.14694/) (2022)
- [Online Incident Response Planning under Model Misspecification through Bayesian Learning and Belief Quantization](https://mlsanity.com/paper/2508.14385/) (2025)
- [Scalable Policy-Based RL Algorithms for POMDPs](https://mlsanity.com/paper/2510.06540/) (2025)
- [Reinforcement Learning in Switching Non-Stationary Markov Decision Processes: Algorithms and Convergence Analysis](https://mlsanity.com/paper/2503.18607/) (2025)
- [Data-Assimilated Model-Based Reinforcement Learning for Partially Observed Chaotic Flows](https://mlsanity.com/paper/2504.16588/) (2025)
- [Learning Optimal Defender Strategies for CAGE-2 using a POMDP Model](https://mlsanity.com/paper/2509.06539/) (2025)
- [Geometric Learning of Hidden Markov Models via a Method of Moments Algorithm](https://mlsanity.com/paper/2207.00818/) (2022)
- [Distributed Bayesian Learning of Dynamic States](https://mlsanity.com/paper/2212.02565/) (2022)
- [CLT and Edgeworth Expansion for m-out-of-n Bootstrap Estimators of The Studentized Median](https://mlsanity.com/paper/2505.11725/) (2025)
- [A Teacher-Student Markov Decision Process-based Framework for Online Correctional Learning](https://mlsanity.com/paper/2111.07818/) (2021)
- [How can a Radar Mask its Cognition?](https://mlsanity.com/paper/2210.11444/) (2022)
- [Policy Evaluation in Decentralized POMDPs with Belief Sharing](https://mlsanity.com/paper/2302.04151/) (2023)
- [Solving Truly Massive Budgeted Monotonic POMDPs with Oracle-Guided Meta-Reinforcement Learning](https://mlsanity.com/paper/2408.07192/) (2024)
- [Efficient Learning of POMDPs with Known Observation Model in Average-Reward Setting](https://mlsanity.com/paper/2410.01331/) (2024)
- [Data-assimilated model-informed reinforcement learning](https://mlsanity.com/paper/2506.01755/) (2025)
- [Knowing When to Ask for Help: Bayesian Self-Escalation in Hierarchical LLM Agents](https://mlsanity.com/paper/2608.24087/) (2026)
- [Langevin Dynamics for Adaptive Inverse Reinforcement Learning of Stochastic Gradient Algorithms](https://mlsanity.com/paper/2006.11674/) (2020)
- [Cell Multi-Bernoulli (Cell-MB) Sensor Control for Multi-object Search-While-Tracking (SWT)](https://mlsanity.com/paper/2108.11236/) (2021)
- [Achieving widetildeO(√T) Regret in Average-Reward POMDPs with Known Observation Models](https://mlsanity.com/paper/2501.18790/) (2025)
- [Inferring Group Intent as a Cooperative Game. An NLP-based Framework for Trajectory Analysis](https://mlsanity.com/paper/2510.23905/) (2025)

## Abstract (arXiv)

This document is an internet supplement to my book "Partially Observed Markov Decision Processes - From Filtering to Controlled Sensing" published by Cambridge University Press in 2016. This internet supplement contains exercises, examples and case studies. The material appears in this internet supplement (instead of the book) so that it can be updated. This document will evolve over time and further discussion and examples will be added. This internet supplement document is work in progress and will be updated periodically. I welcome constructive comments from readers of the book and this internet supplement.