# Bridging POMDPs and Bayesian decision making for robust maintenance planning under model uncertainty: An application to railway systems

arXiv: https://arxiv.org/abs/2212.07933 · HTML: https://mlsanity.com/paper/2212.07933/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 62 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 10 of 10)

- [On Improving Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1804.06309/) (2018)
- [Managing engineering systems with large state and action spaces through deep reinforcement learning](https://mlsanity.com/paper/1811.02052/) (2018)
- [Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints](https://mlsanity.com/paper/2007.01380/) (2020)
- [An On-Line POMDP Solver for Continuous Observation Spaces](https://mlsanity.com/paper/2011.02076/) (2020)
- [On risk-based active learning for structural health monitoring](https://mlsanity.com/paper/2105.05622/) (2021)
- [Optimal Inspection and Maintenance Planning for Deteriorating Structural Components through Dynamic Bayesian Networks and Markov Decision Processes](https://mlsanity.com/paper/2009.04547/) (2020)
- [Inference and dynamic decision-making for deteriorating systems with probabilistic dependencies through Bayesian networks and deep reinforcement learning](https://mlsanity.com/paper/2209.01092/) (2022)
- [Value of information from vibration-based structural health monitoring extracted via Bayesian model updating](https://mlsanity.com/paper/2103.07382/) (2021)
- [Which Model to Trust: Assessing the Influence of Models on the Performance of Reinforcement Learning Algorithms for Continuous Control Tasks](https://mlsanity.com/paper/2110.13079/) (2021)
- [Autoregressive Hidden Markov Models with partial knowledge on latent space applied to aero-engines prognostics](https://mlsanity.com/paper/2105.00211/) (2021)

## Built on by (most cited first, 11 of 11)

- [POMDP inference and robust solution via deep reinforcement learning: An application to railway optimal maintenance](https://mlsanity.com/paper/2307.08082/) (2023)
- [A digital twin framework for civil engineering structures](https://mlsanity.com/paper/2308.01445/) (2023)
- [Multi-agent deep reinforcement learning with centralized training and decentralized execution for transportation infrastructure management](https://mlsanity.com/paper/2401.12455/) (2024)
- [Quantifying the value of information transfer in population-based SHM](https://mlsanity.com/paper/2311.03083/) (2023)
- [A local squared Wasserstein-2 method for efficient reconstruction of models with uncertainty](https://mlsanity.com/paper/2406.06825/) (2024)
- [Deep Belief Markov Models for POMDP Inference](https://mlsanity.com/paper/2503.13438/) (2025)
- [A generative adversarial network optimization method for damage detection and digital twinning by deep AI fault learning: Z24 Bridge structural health monitoring benchmark validation](https://mlsanity.com/paper/2511.00099/) (2025)
- [Real-Time Structural Health Monitoring with Bayesian Neural Networks: Distinguishing Aleatoric and Epistemic Uncertainty for Digital Twin Frameworks](https://mlsanity.com/paper/2512.03115/) (2025)
- [Adaptive digital twins for predictive decision-making: Online Bayesian learning of transition dynamics](https://mlsanity.com/paper/2512.13919/) (2025)
- [Optimal sequential decision-making for error propagation mitigation in digital twins](https://mlsanity.com/paper/2604.22168/) (2026)
- [Graph-Based Inference and Topology-Aware Multi-Agent Reinforcement Learning for Large-Scale Railway Network Management](https://mlsanity.com/paper/2609.30150/) (2026)

## Abstract (arXiv)

Structural Health Monitoring (SHM) describes a process for inferring quantifiable metrics of structural condition, which can serve as input to support decisions on the operation and maintenance of infrastructure assets. Given the long lifespan of critical structures, this problem can be cast as a sequential decision making problem over prescribed horizons. Partially Observable Markov Decision Processes (POMDPs) offer a formal framework to solve the underlying optimal planning task. However, two issues can undermine the POMDP solutions. Firstly, the need for a model that can adequately describe the evolution of the structural condition under deterioration or corrective actions and, secondly, the non-trivial task of recovery of the observation process parameters from available monitoring data. Despite these potential challenges, the adopted POMDP models do not typically account for uncertainty on model parameters, leading to solutions which can be unrealistically confident. In this work, we address both key issues. We present a framework to estimate POMDP transition and observation model parameters directly from available data, via Markov Chain Monte Carlo (MCMC) sampling of a Hidden Markov Model (HMM) conditioned on actions. The MCMC inference estimates distributions of the involved model parameters. We then form and solve the POMDP problem by exploiting the inferred distributions, to derive solutions that are robust to model uncertainty. We successfully apply our approach on maintenance planning for railway track assets on the basis of a "fractal value" indicator, which is computed from actual railway monitoring data.