# Managing engineering systems with large state and action spaces through deep reinforcement learning

arXiv: https://arxiv.org/abs/1811.02052 · HTML: https://mlsanity.com/paper/1811.02052/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2018.

## Built on by (most cited first, 30 of 32)

- [MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer](https://mlsanity.com/paper/2206.10607/) (2022)
- [POMDP inference and robust solution via deep reinforcement learning: An application to railway optimal maintenance](https://mlsanity.com/paper/2307.08082/) (2023)
- [Inference and dynamic decision-making for deteriorating systems with probabilistic dependencies through Bayesian networks and deep reinforcement learning](https://mlsanity.com/paper/2209.01092/) (2022)
- [Bridging POMDPs and Bayesian decision making for robust maintenance planning under model uncertainty: An application to railway systems](https://mlsanity.com/paper/2212.07933/) (2022)
- [IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARL](https://mlsanity.com/paper/2306.11551/) (2023)
- [A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.00451/) (2023)
- [Multi-agent deep reinforcement learning with centralized training and decentralized execution for transportation infrastructure management](https://mlsanity.com/paper/2401.12455/) (2024)
- [Reinforcement Learning In Two Player Zero Sum Simultaneous Action Games](https://mlsanity.com/paper/2110.04835/) (2021)
- [Opinion-Guided Reinforcement Learning](https://mlsanity.com/paper/2405.17287/) (2024)
- [Design and Planning of Flexible Mobile Micro-Grids Using Deep Reinforcement Learning](https://mlsanity.com/paper/2212.04136/) (2022)
- [Hierarchical Task Network Planning for Facilitating Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.08359/) (2023)
- [An investigation of belief-free DRL and MCTS for inspection and maintenance planning](https://mlsanity.com/paper/2312.14824/) (2023)
- [Deep Multi-Objective Reinforcement Learning for Utility-Based Infrastructural Maintenance Optimization](https://mlsanity.com/paper/2406.06184/) (2024)
- [InfraLib: Enabling Reinforcement Learning and Decision-Making for Large-Scale Infrastructure Management](https://mlsanity.com/paper/2409.03167/) (2024)
- [Resource Governance in Networked Systems via Integrated Variational Autoencoders and Reinforcement Learning](https://mlsanity.com/paper/2410.23393/) (2024)
- [A generative adversarial network optimization method for damage detection and digital twinning by deep AI fault learning: Z24 Bridge structural health monitoring benchmark validation](https://mlsanity.com/paper/2511.00099/) (2025)
- [Distributional Reinforcement Learning for Condition-Based Maintenance of Multi-Pump Equipment](https://mlsanity.com/paper/2602.00051/) (2026)
- [Information Avoidance and Overvaluation in Sequential Decision Making under Epistemic Constraints](https://mlsanity.com/paper/2106.04984/) (2021)
- [A Decentralized Communication Framework based on Dual-Level Recurrence for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2202.10612/) (2022)
- [Farm-wide virtual load monitoring for offshore wind structures via Bayesian neural networks](https://mlsanity.com/paper/2211.00642/) (2022)
- [An Advantage-based Optimization Method for Reinforcement Learning in Large Action Space](https://mlsanity.com/paper/2412.12605/) (2024)
- [Convex Is Back: Solving Belief MDPs With Convexity-Informed Deep Reinforcement Learning](https://mlsanity.com/paper/2502.09298/) (2025)
- [Dual-Individual Genetic Algorithm: A Dual-Individual Approach for Efficient Training of Multi-Layer Neural Networks](https://mlsanity.com/paper/2504.17346/) (2025)
- [Decoupled Hierarchical Reinforcement Learning with State Abstraction for Discrete Grids](https://mlsanity.com/paper/2506.02050/) (2025)
- [A Generative Physics-Informed Reinforcement Learning-Based Approach for Construction of Representative Drive Cycle](https://mlsanity.com/paper/2506.07929/) (2025)
- [AutoIndexer: A Reinforcement Learning-Enhanced Index Advisor Towards Scaling Workloads](https://mlsanity.com/paper/2507.23084/) (2025)
- [A convolutional neural network deep learning method for model class selection](https://mlsanity.com/paper/2511.03743/) (2025)
- [Adaptive digital twins for predictive decision-making: Online Bayesian learning of transition dynamics](https://mlsanity.com/paper/2512.13919/) (2025)
- [COOL-MC: Verifying and Explaining RL Policies for Multi-bridge Network Maintenance](https://mlsanity.com/paper/2603.07546/) (2026)
- [Interpretable Deep Reinforcement Learning for Element-level Bridge Life-cycle Optimization](https://mlsanity.com/paper/2604.02528/) (2026)

## Abstract (arXiv)

Decision-making for engineering systems can be efficiently formulated as a Markov Decision Process (MDP) or a Partially Observable MDP (POMDP). Typical MDP and POMDP solution procedures utilize offline knowledge about the environment and provide detailed policies for relatively small systems with tractable state and action spaces. However, in large multi-component systems the sizes of these spaces easily explode, as system states and actions scale exponentially with the number of components, whereas environment dynamics are difficult to be described in explicit forms for the entire system and may only be accessible through numerical simulators. In this work, to address these issues, an integrated Deep Reinforcement Learning (DRL) framework is introduced. The Deep Centralized Multi-agent Actor Critic (DCMAC) is developed, an off-policy actor-critic DRL approach, providing efficient life-cycle policies for large multi-component systems operating in high-dimensional spaces. Apart from deep function approximations that parametrize large state spaces, DCMAC also adopts a factorized representation of the system actions, being able to designate individualized component- and subsystem-level decisions, while maintaining a centralized value function for the entire system. DCMAC compares well against Deep Q-Network (DQN) solutions and exact policies, where applicable, and outperforms optimized baselines that are based on time-based, condition-based and periodic policies.