# Development of Deep Transformer-Based Models for Long-Term Prediction of Transient Production of Oil Wells

arXiv: https://arxiv.org/abs/2110.06059 · HTML: https://mlsanity.com/paper/2110.06059/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [A Predictive Model for Steady-State Multiphase Pipe Flow: Machine Learning on Lab Data](https://mlsanity.com/paper/1905.09746/) (2019)
- [Data-driven model for hydraulic fracturing design optimization. Part II: Inverse problem](https://mlsanity.com/paper/2108.00751/) (2021)

## Built on by (most cited first, 3 of 3)

- [Self-supervised similarity models based on well-logging data](https://mlsanity.com/paper/2209.12444/) (2022)
- [Hidden Leaks in Time Series Forecasting: How Data Leakage Affects LSTM Evaluation Across Configurations and Validation Strategies](https://mlsanity.com/paper/2512.06932/) (2025)
- [Robust representations of oil wells' intervals via sparse attention mechanism](https://mlsanity.com/paper/2212.14246/) (2022)

## Abstract (arXiv)

We propose a novel approach to data-driven modeling of a transient production of oil wells. We apply the transformer-based neural networks trained on the multivariate time series composed of various parameters of oil wells measured during their exploitation. By tuning the machine learning models for a single well (ignoring the effect of neighboring wells) on the open-source field datasets, we demonstrate that transformer outperforms recurrent neural networks with LSTM/GRU cells in the forecasting of the bottomhole pressure dynamics. We apply the transfer learning procedure to the transformer-based surrogate model, which includes the initial training on the dataset from a certain well and additional tuning of the model's weights on the dataset from a target well. Transfer learning approach helps to improve the prediction capability of the model. Next, we generalize the single-well model based on the transformer architecture for multiple wells to simulate complex transient oilfield-level patterns. In other words, we create the global model which deals with the dataset, comprised of the production history from multiple wells, and allows for capturing the well interference resulting in more accurate prediction of the bottomhole pressure or flow rate evolutions for each well under consideration. The developed instruments for a single-well and oilfield-scale modelling can be used to optimize the production process by selecting the operating regime and submersible equipment to increase the hydrocarbon recovery. In addition, the models can be helpful to perform well-testing avoiding costly shut-in operations.