Safe Policy Improvement for POMDPs via Finite-State Controllers
Not on the map. Why?
Abstract (arXiv)
We study safe policy improvement (SPI) for partially observable Markov decision processes (POMDPs). SPI is an offline reinforcement learning (RL) problem that assumes access to (1) historical data about an environment, and (2) the so-called behavior policy that previously generated this data by interacting with the environment. SPI methods neither require access to a model nor the environment itself, and aim to reliably improve the behavior policy in an offline manner. Existing methods make the strong assumption that the environment is fully observable. In our novel approach to the SPI problem for POMDPs, we assume that a finite-state controller (FSC) represents the behavior policy and that finite memory is sufficient to derive optimal policies. This assumption allows us to map the POMDP to a finite-state fully observable MDP, the history MDP. We estimate this MDP by combining the historical data and the memory of the FSC, and compute an improved policy using an off-the-shelf SPI algorithm. The underlying SPI method constrains the policy-space according to the available data, such that the newly computed policy only differs from the behavior policy when sufficient data was available. We show that this new policy, converted into a new FSC for the (unknown) POMDP, outperforms the behavior policy with high probability. Experimental results on several well-established benchmarks show the applicability of the approach, even in cases where finite memory is not sufficient.
Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 17 of its 47 references that are in the map. First posted on arXiv 12 Jan 2023.
Builds on
17 of 47 references.
References · 17
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
- 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
- 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
- 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
- 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
- 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
- 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
- 2013Solving POMDPs by Searching the Space of Finite Policiescited by 25
- 2020Robust Finite-State Controllers for Uncertain POMDPscited by 23
- 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
- 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPscited by 22
5 more
- 2013Learning Finite-State Controllers for Partially Observable Environmentscited by 18
- 2020Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraintscited by 18
- 2019Safe Policy Improvement with an Estimated Baseline Policycited by 16
- 2022Safe Policy Improvement Approaches on Discrete Markov Decision Processescited by 5
- 2022Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learningcited by 4
Built on by
Citing papers per year: 2023: 3; 2024: 2; 2025: 1; 2026: 4.
10 papers.
- 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meetcited by 18
- 2023Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuringcited by 10
- 2023More for Less: Safe Policy Improvement With Stronger Performance Guaranteescited by 9
- 2023Decision-Making Under Uncertainty: Beyond Probabilitiescited by 8
- 2024Learning Explainable and Better Performing Representations of POMDP Strategiescited by 4
- 2026Formally Verifying and Explaining Sepsis Treatment Policies with COOL-MCcited by 3
- 2025Data-Efficient Safe Policy Improvement Using Parametric Structurecited by 2
- 2026COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Managementcited by 0
- 2026When Does Action Credit Need Updating?cited by 0
- 2026Bellman-Certified Rounding for Sparse Policy Deployment in MDPscited by 0
- 2026Bellman-Certified Rounding for Sparse Policy Deployment in MDPs29 Sep 2026
- 2026When Does Action Credit Need Updating?24 Sep 2026
- 2026COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Management2 Mar 2026
- 2026Formally Verifying and Explaining Sepsis Treatment Policies with COOL-MC16 Feb 2026
- 2025Data-Efficient Safe Policy Improvement Using Parametric Structure21 Jul 2025
- 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet18 Nov 2024
- 2024Learning Explainable and Better Performing Representations of POMDP Strategies15 Jan 2024
- 2023More for Less: Safe Policy Improvement With Stronger Performance Guarantees13 May 2023
- 2023Act-Then-Measure: Reinforcement Learning for Partially Observable Environments with Active Measuring14 Mar 2023
- 2023Decision-Making Under Uncertainty: Beyond Probabilities10 Mar 2023
Similar, not cited
Close in wording, no citation either way
- 2022Safe Policy Improvement Approaches and their Limitations
- 2024Decision-Point Guided Safe Policy Improvement
- 2025Deep SPI: Safe Policy Improvement via World Models
- 2026Robust Probabilistic Shielding for Safe Offline Reinforcement Learning
- 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
- 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes
- 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
- 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning