# POMDP inference and robust solution via deep reinforcement learning: An application to railway optimal maintenance

arXiv: https://arxiv.org/abs/2307.08082 · HTML: https://mlsanity.com/paper/2307.08082/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 49 references that are in the map.
- First posted on arXiv 16 Jul 2023.

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [On Improving Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1804.06309/) (2018)
- [Managing engineering systems with large state and action spaces through deep reinforcement learning](https://mlsanity.com/paper/1811.02052/) (2018)
- [Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints](https://mlsanity.com/paper/2007.01380/) (2020)
- [Optimal Inspection and Maintenance Planning for Deteriorating Structural Components through Dynamic Bayesian Networks and Markov Decision Processes](https://mlsanity.com/paper/2009.04547/) (2020)
- [Inference and dynamic decision-making for deteriorating systems with probabilistic dependencies through Bayesian networks and deep reinforcement learning](https://mlsanity.com/paper/2209.01092/) (2022)
- [Bridging POMDPs and Bayesian decision making for robust maintenance planning under model uncertainty: An application to railway systems](https://mlsanity.com/paper/2212.07933/) (2022)
- [Which Model to Trust: Assessing the Influence of Models on the Performance of Reinforcement Learning Algorithms for Continuous Control Tasks](https://mlsanity.com/paper/2110.13079/) (2021)

## Built on by (most cited first, 13 of 13)

- [Multi-agent deep reinforcement learning with centralized training and decentralized execution for transportation infrastructure management](https://mlsanity.com/paper/2401.12455/) (2024)
- [Deep Belief Markov Models for POMDP Inference](https://mlsanity.com/paper/2503.13438/) (2025)
- [An investigation of belief-free DRL and MCTS for inspection and maintenance planning](https://mlsanity.com/paper/2312.14824/) (2023)
- [Dynamic Pricing in High-Speed Railways Using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2501.08234/) (2025)
- [Convex Is Back: Solving Belief MDPs With Convexity-Informed Deep Reinforcement Learning](https://mlsanity.com/paper/2502.09298/) (2025)
- [A Multi-Agent Reinforcement Learning Approach for Cooperative Air-Ground-Human Crowdsensing in Emergency Rescue](https://mlsanity.com/paper/2505.06997/) (2025)
- [Success in Humanoid Reinforcement Learning under Partial Observation](https://mlsanity.com/paper/2507.18883/) (2025)
- [Hierarchical Deep Reinforcement Learning Framework for Multi-Year Asset Management Under Budget Constraints](https://mlsanity.com/paper/2507.19458/) (2025)
- [Memoryless Policy Iteration for Episodic POMDPs](https://mlsanity.com/paper/2512.11082/) (2025)
- [Adaptive digital twins for predictive decision-making: Online Bayesian learning of transition dynamics](https://mlsanity.com/paper/2512.13919/) (2025)
- [From Ambiguity to Action: A POMDP Perspective on Partial Multi-Label Ambiguity and Its Horizon-One Resolution](https://mlsanity.com/paper/2602.04255/) (2026)
- [COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenance](https://mlsanity.com/paper/2603.07546/) (2026)
- [Graph-Based Inference and Topology-Aware Multi-Agent Reinforcement Learning for Large-Scale Railway Network Management](https://mlsanity.com/paper/2609.30150/) (2026)

## Abstract (arXiv)

Partially Observable Markov Decision Processes (POMDPs) can model complex sequential decision-making problems under stochastic and uncertain environments. A main reason hindering their broad adoption in real-world applications is the lack of availability of a suitable POMDP model or a simulator thereof. Available solution algorithms, such as Reinforcement Learning (RL), require the knowledge of the transition dynamics and the observation generating process, which are often unknown and non-trivial to infer. In this work, we propose a combined framework for inference and robust solution of POMDPs via deep RL. First, all transition and observation model parameters are jointly inferred via Markov Chain Monte Carlo sampling of a hidden Markov model, which is conditioned on actions, in order to recover full posterior distributions from the available data. The POMDP with uncertain parameters is then solved via deep RL techniques with the parameter distributions incorporated into the solution via domain randomization, in order to develop solutions that are robust to model uncertainty. As a further contribution, we compare the use of transformers and long short-term memory networks, which constitute model-free RL solutions, with a model-based/model-free hybrid approach. We apply these methods to the real-world problem of optimal maintenance planning for railway assets.