# Welfare Maximization Algorithm for Solving Budget-Constrained Multi-Component POMDPs

arXiv: https://arxiv.org/abs/2303.10302 · HTML: https://mlsanity.com/paper/2303.10302/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 29 references that are in the map.
- First posted on arXiv 18 Mar 2023.

## Builds on (most cited first, 6 of 6)

- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes](https://mlsanity.com/paper/1302.1525/) (2013)
- [Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints](https://mlsanity.com/paper/2007.01380/) (2020)
- [Solving POMDPs by Searching in Policy Space](https://mlsanity.com/paper/1301.7380/) (2013)
- [Optimal Inspection and Maintenance Planning for Deteriorating Structural Components through Dynamic Bayesian Networks and Markov Decision Processes](https://mlsanity.com/paper/2009.04547/) (2020)
- [Ranking and Selection as Stochastic Control](https://mlsanity.com/paper/1710.02619/) (2017)

## Built on by (most cited first, 5 of 5)

- [ComTraQ-MPC: Meta-Trained DQN-MPC Integration for Trajectory Tracking with Limited Active Localization Updates](https://mlsanity.com/paper/2403.01564/) (2024)
- [Resilient Constrained Reinforcement Learning](https://mlsanity.com/paper/2312.17194/) (2023)
- [Solving Truly Massive Budgeted Monotonic POMDPs with Oracle-Guided Meta-Reinforcement Learning](https://mlsanity.com/paper/2408.07192/) (2024)
- [InfraLib: Enabling Reinforcement Learning and Decision-Making for Large-Scale Infrastructure Management](https://mlsanity.com/paper/2409.03167/) (2024)
- [Capacity-Aware Planning and Scheduling in Budget-Constrained Multi-Agent MDPs: A Meta-RL Approach](https://mlsanity.com/paper/2410.21249/) (2024)

## Abstract (arXiv)

Partially Observable Markov Decision Processes (POMDPs) provide an efficient way to model real-world sequential decision making processes. Motivated by the problem of maintenance and inspection of a group of infrastructure components with independent dynamics, this paper presents an algorithm to find the optimal policy for a multi-component budget-constrained POMDP. We first introduce a budgeted-POMDP model (b-POMDP) which enables us to find the optimal policy for a POMDP while adhering to budget constraints. Next, we prove that the value function or maximal collected reward for a b-POMDP is a concave function of the budget for the finite horizon case. Our second contribution is an algorithm to calculate the optimal policy for a multi-component budget-constrained POMDP by finding the optimal budget split among the individual component POMDPs. The optimal budget split is posed as a welfare maximization problem and the solution is computed by exploiting the concave nature of the value function. We illustrate the effectiveness of the proposed algorithm by proposing a maintenance and inspection policy for a group of real-world infrastructure components with different deterioration dynamics, inspection and maintenance costs. We show that the proposed algorithm vastly outperforms the policy currently used in practice.