# Deep reinforcement learning for optimal well control in subsurface systems with uncertain geology

arXiv: https://arxiv.org/abs/2203.13375 · HTML: https://mlsanity.com/paper/2203.13375/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 57 references that are in the map.
- First posted on arXiv 24 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification](https://mlsanity.com/paper/1801.06879/) (2018)
- [A deep-learning-based surrogate model for data assimilation in dynamic subsurface flow problems](https://mlsanity.com/paper/1908.05823/) (2019)
- [Direct shape optimization through deep reinforcement learning](https://mlsanity.com/paper/1908.09885/) (2019)
- [Theory-guided Auto-Encoder for Surrogate Construction and Inverse Modeling](https://mlsanity.com/paper/2011.08618/) (2020)
- [A Deep-Learning-Based Geological Parameterization for History Matching Complex Models](https://mlsanity.com/paper/1807.02716/) (2018)
- [Deep Reinforcement Learning for Constrained Field Development Optimization in Subsurface Two-phase Flow](https://mlsanity.com/paper/2104.00527/) (2021)
- [Use of low-fidelity models with machine-learning error correction for well placement optimization](https://mlsanity.com/paper/2111.02960/) (2021)
- [Convolutional-Recurrent Neural Network Proxy for Robust Optimization and Closed-Loop Reservoir Management](https://mlsanity.com/paper/2203.07524/) (2022)

## Built on by (most cited first, 9 of 9)

- [Transformers in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.05979/) (2023)
- [Graph Network Surrogate Model for Subsurface Flow Optimization](https://mlsanity.com/paper/2312.08625/) (2023)
- [Optimal Sequential Decision-Making in Geosteering: A Reinforcement Learning Approach](https://mlsanity.com/paper/2310.04772/) (2023)
- [Multi-Asset Closed-Loop Reservoir Management Using Deep Reinforcement Learning](https://mlsanity.com/paper/2207.10376/) (2022)
- [Reinforcement learning based data assimilation for unknown state model](https://mlsanity.com/paper/2511.02286/) (2025)
- [The future of AI in critical mineral exploration](https://mlsanity.com/paper/2512.02879/) (2025)
- [Optimization of geological carbon storage operations with multimodal latent dynamic model and deep reinforcement learning](https://mlsanity.com/paper/2406.04575/) (2024)
- [Closed-Loop CO2 Storage Control With History-Based Reinforcement Learning and Latent Model-Based Adaptation](https://mlsanity.com/paper/2605.02405/) (2026)
- [Efficient Geothermal Well-Control Optimization via Diffusion-Surrogate Reinforcement Learning](https://mlsanity.com/paper/2608.28791/) (2026)

## Abstract (arXiv)

A general control policy framework based on deep reinforcement learning (DRL) is introduced for closed-loop decision making in subsurface flow settings. Traditional closed-loop modeling workflows in this context involve the repeated application of data assimilation/history matching and robust optimization steps. Data assimilation can be particularly challenging in cases where both the geological style (scenario) and individual model realizations are uncertain. The closed-loop reservoir management (CLRM) problem is formulated here as a partially observable Markov decision process, with the associated optimization problem solved using a proximal policy optimization algorithm. This provides a control policy that instantaneously maps flow data observed at wells (as are available in practice) to optimal well pressure settings. The policy is represented by a temporal convolution and gated transformer blocks. Training is performed in a preprocessing step with an ensemble of prior geological models, which can be drawn from multiple geological scenarios. Example cases involving the production of oil via water injection, with both 2D and 3D geological models, are presented. The DRL-based methodology is shown to result in an NPV increase of 15% (for the 2D cases) and 33% (3D cases) relative to robust optimization over prior models, and to an average improvement of 4% in NPV relative to traditional CLRM. The solutions from the control policy are found to be comparable to those from deterministic optimization, in which the geological model is assumed to be known, even when multiple geological scenarios are considered. The control policy approach results in a 76% decrease in computational cost relative to traditional CLRM with the algorithms and parameter settings considered in this work.