# ProcessTransformer: Predictive Business Process Monitoring with Transformer Network

arXiv: https://arxiv.org/abs/2104.00721 · HTML: https://mlsanity.com/paper/2104.00721/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Predictive Business Process Monitoring with LSTM Neural Networks](https://mlsanity.com/paper/1612.02130/) (2016)
- [Entity Embeddings of Categorical Variables](https://mlsanity.com/paper/1604.06737/) (2016)
- [Deep Learning for Predictive Business Process Monitoring: Review and Benchmark](https://mlsanity.com/paper/2009.13251/) (2020)
- [Predictive Business Process Monitoring via Generative Adversarial Nets: The Case of Next Event Prediction](https://mlsanity.com/paper/2003.11268/) (2020)
- [Time and Activity Sequence Prediction of Business Process Instances](https://mlsanity.com/paper/1602.07566/) (2016)
- [LSTM Networks for Data-Aware Remaining Time Prediction of Business Process Instances](https://mlsanity.com/paper/1711.03822/) (2017)
- [Time Matters: Time-Aware LSTMs for Predictive Business Process Monitoring](https://mlsanity.com/paper/2010.00889/) (2020)
- [An empirical comparison of deep-neural-network architectures for next activity prediction using context-enriched process event logs](https://mlsanity.com/paper/2005.01194/) (2020)
- [Encoder-Decoder Generative Adversarial Nets for Suffix Generation and Remaining Time Prediction of Business Process Models](https://mlsanity.com/paper/2007.16030/) (2020)

## Built on by (most cited first, 30 of 37)

- [PGTNet: A Process Graph Transformer Network for Remaining Time Prediction of Business Process Instances](https://mlsanity.com/paper/2404.06267/) (2024)
- [Embedding Graph Convolutional Networks in Recurrent Neural Networks for Predictive Monitoring](https://mlsanity.com/paper/2112.09641/) (2021)
- [What Averages Do Not Tell -- Predicting Real Life Processes with Sequential Deep Learning](https://mlsanity.com/paper/2110.10225/) (2021)
- [Building Interpretable Models for Business Process Prediction using Shared and Specialised Attention Mechanisms](https://mlsanity.com/paper/2109.01419/) (2021)
- [Generating Privacy-Preserving Process Data with Deep Generative Models](https://mlsanity.com/paper/2203.07949/) (2022)
- [Can recurrent neural networks learn process model structure?](https://mlsanity.com/paper/2212.06430/) (2022)
- [Directly Follows Graphs Go Predictive Process Monitoring With Graph Neural Networks](https://mlsanity.com/paper/2503.03197/) (2025)
- [Explainable Artificial Intelligence for Improved Modeling of Processes](https://mlsanity.com/paper/2212.00695/) (2022)
- [A Survey on Event Prediction Methods from a Systems Perspective: Bringing Together Disparate Research Areas](https://mlsanity.com/paper/2302.04018/) (2023)
- [Attention Please: What Transformer Models Really Learn for Process Prediction](https://mlsanity.com/paper/2408.07097/) (2024)
- [An Innovative Next Activity Prediction Using Process Entropy and Dynamic Attribute-Wise-Transformer in Predictive Business Process Monitoring](https://mlsanity.com/paper/2502.10573/) (2025)
- [Time-Aware and Transition-Semantic Graph Neural Networks for Interpretable Predictive Business Process Monitoring](https://mlsanity.com/paper/2508.09527/) (2025)
- [Can deep neural networks learn process model structure? An assessment framework and analysis](https://mlsanity.com/paper/2202.11985/) (2022)
- [Enhancing Stochastic Petri Net-based Remaining Time Prediction using k-Nearest Neighbors](https://mlsanity.com/paper/2206.13109/) (2022)
- [RLHGNN: Reinforcement Learning-driven Heterogeneous Graph Neural Network for Next Activity Prediction in Business Processes](https://mlsanity.com/paper/2507.02690/) (2025)
- [Predictive Process Monitoring Using Object-centric Graph Embeddings](https://mlsanity.com/paper/2507.15411/) (2025)
- [From Source to Target: Leveraging Transfer Learning for Predictive Process Monitoring in Organizations](https://mlsanity.com/paper/2508.08061/) (2025)
- [Neuro-Symbolic Learning for Predictive Process Monitoring via Two-Stage Logic Tensor Networks with Rule Pruning](https://mlsanity.com/paper/2603.26944/) (2026)
- [Compliance-Aware Predictive Process Monitoring: A Neuro-Symbolic Approach](https://mlsanity.com/paper/2603.26948/) (2026)
- [Universal Multi-Modal Traceformer: Integrating Heterogeneous Context for Process Event Prediction](https://mlsanity.com/paper/2609.24579/) (2026)
- [Analysis of business process automation as linear time-invariant system network](https://mlsanity.com/paper/2302.04785/) (2023)
- [Structural Positional Encoding for knowledge integration in transformer-based medical process monitoring](https://mlsanity.com/paper/2403.08836/) (2024)
- [Anomaly Correction of Business Processes Using Transformer Autoencoder](https://mlsanity.com/paper/2404.10211/) (2024)
- [Extending predictive process monitoring for collaborative processes](https://mlsanity.com/paper/2409.09212/) (2024)
- [Rethinking BPS: A Utility-Based Evaluation Framework](https://mlsanity.com/paper/2505.22316/) (2025)
- [Leveraging Data Augmentation and Siamese Learning for Predictive Process Monitoring](https://mlsanity.com/paper/2507.18293/) (2025)
- [On the Simplification of Neural Network Architectures for Predictive Process Monitoring](https://mlsanity.com/paper/2509.17145/) (2025)
- [Towards Reproducibility in Predictive Process Mining: SPICE -- A Deep Learning Library](https://mlsanity.com/paper/2512.16715/) (2025)
- [Promoting Simple Agents: Ensemble Methods for Event-Log Prediction](https://mlsanity.com/paper/2604.21629/) (2026)
- [David vs. Goliath in Next Activity Prediction: Argmax vs. LSTM, Transformer, and LLM](https://mlsanity.com/paper/2606.15868/) (2026)

## Abstract (arXiv)

Predictive business process monitoring focuses on predicting future characteristics of a running process using event logs. The foresight into process execution promises great potentials for efficient operations, better resource management, and effective customer services. Deep learning-based approaches have been widely adopted in process mining to address the limitations of classical algorithms for solving multiple problems, especially the next event and remaining-time prediction tasks. Nevertheless, designing a deep neural architecture that performs competitively across various tasks is challenging as existing methods fail to capture long-range dependencies in the input sequences and perform poorly for lengthy process traces. In this paper, we propose ProcessTransformer, an approach for learning high-level representations from event logs with an attention-based network. Our model incorporates long-range memory and relies on a self-attention mechanism to establish dependencies between a multitude of event sequences and corresponding outputs. We evaluate the applicability of our technique on nine real event logs. We demonstrate that the transformer-based model outperforms several baselines of prior techniques by obtaining on average above 80% accuracy for the task of predicting the next activity. Our method also perform competitively, compared to baselines, for the tasks of predicting event time and remaining time of a running case