mlsanityData to 14 Oct 2026

Hierarchical Deep Reinforcement Learning Framework for Multi-Year Asset Management Under Budget Constraints

arXiv 2507.19458 ↗First posted 25 Jul 2025cs.AI cs.LG cs.SYcited by 0

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

Budget planning and maintenance optimization are crucial for infrastructure asset management, ensuring cost-effectiveness and sustainability. However, the complexity arising from combinatorial action spaces, diverse asset deterioration, stringent budget constraints, and environmental uncertainty significantly limits existing methods' scalability. This paper proposes a Hierarchical Deep Reinforcement Learning methodology specifically tailored to multi-year infrastructure planning. Our approach decomposes the problem into two hierarchical levels: a high-level Budget Planner allocating annual budgets within explicit feasibility bounds, and a low-level Maintenance Planner prioritizing assets within the allocated budget. By structurally separating macro-budget decisions from asset-level prioritization and integrating linear programming projection within a hierarchical Soft Actor-Critic framework, the method efficiently addresses exponential growth in the action space and ensures rigorous budget compliance. A case study evaluating sewer networks of varying sizes (10, 15, and 20 sewersheds) illustrates the effectiveness of the proposed approach. Compared to conventional Deep Q-Learning and enhanced genetic algorithms, our methodology converges more rapidly, scales effectively, and consistently delivers near-optimal solutions even as network size grows.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 10 of its 59 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

10 of 59 references.

References · 10

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  5. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  6. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  7. 2020FACMAC: Factored Multi-Agent Centralised Policy Gradientscited by 178
  8. 2020Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraintscited by 18
  9. 2023POMDP inference and robust solution via deep reinforcement learning: An application to railway optimal maintenancecited by 13
  10. 2022Mean-Field Approximation of Cooperative Constrained Multi-Agent Reinforcement Learning (CMARL)cited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Year Maintenance Planning for Large-Scale Infrastructure Systems: A Novel Network Deep Q-Learning Approach
  2. 2024Maintenance Strategies for Sewer Pipes with Multi-State Degradation and Deep Reinforcement Learning
  3. 2022A Maintenance Planning Framework using Online and Offline Deep Reinforcement Learning
  4. 2024InfraLib: Enabling Reinforcement Learning and Decision-Making for Large-Scale Infrastructure Management
  5. 2024Maintenance Optimization for Asset Networks with Unknown Degradation Parameters ↗
  6. 2025An approach of deep reinforcement learning for maximizing the net present value of stochastic projects
  7. 2024Multi-agent deep reinforcement learning with centralized training and decentralized execution for transportation infrastructure management
  8. 2026Graph-Based Inference and Topology-Aware Multi-Agent Reinforcement Learning for Large-Scale Railway Network Management