# David vs. Goliath in Next Activity Prediction: Argmax vs. LSTM, Transformer, and LLM

arXiv: https://arxiv.org/abs/2606.15868 · HTML: https://mlsanity.com/paper/2606.15868/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 14 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Predictive Business Process Monitoring with LSTM Neural Networks](https://mlsanity.com/paper/1612.02130/) (2016)
- [Predicting Process Behaviour using Deep Learning](https://mlsanity.com/paper/1612.04600/) (2016)
- [As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages](https://mlsanity.com/paper/2012.05628/) (2020)
- [Deep Learning for Predictive Business Process Monitoring: Review and Benchmark](https://mlsanity.com/paper/2009.13251/) (2020)
- [ProcessTransformer: Predictive Business Process Monitoring with Transformer Network](https://mlsanity.com/paper/2104.00721/) (2021)
- [Predictive Process Monitoring Methods: Which One Suits Me Best?](https://mlsanity.com/paper/1804.02422/) (2018)
- [Creating Unbiased Public Benchmark Datasets with Data Leakage Prevention for Predictive Process Monitoring](https://mlsanity.com/paper/2107.01905/) (2021)
- [Evaluating the Ability of LLMs to Solve Semantics-Aware Process Mining Tasks](https://mlsanity.com/paper/2407.02310/) (2024)
- [Domain Adaptation of LLMs for Process Data](https://mlsanity.com/paper/2509.03161/) (2025)
- [An empirical comparison of deep-neural-network architectures for next activity prediction using context-enriched process event logs](https://mlsanity.com/paper/2005.01194/) (2020)
- [LLMs that Understand Processes: Instruction-tuning for Semantics-Aware Process Mining](https://mlsanity.com/paper/2508.16270/) (2025)

## Abstract (arXiv)

Next activity prediction (NAP) is a cornerstone of predictive process monitoring (PPM), enabling organizations to move from retrospective analysis to proactive process steering. The PPM field has progressed from classical machine learning through deep learning architectures such as LSTMs and Transformers to large language models (LLMs). Despite growing model complexity, no benchmark jointly compares LLMs, Transformers, LSTMs, and simple baselines in a direct sequence modeling setting for NAP. In this paper, we fill this gap with a systematic benchmark. We compare vocabulary-adapted LLMs, Transformers trained from scratch, LLM-distilled Transformers, and LSTMs against a simple counting-based argmax baseline across seven real-life event logs. Our results tell a David vs. Goliath story: pretraining confers no consistent improvement over training from scratch, model size shows little effect on performance, and on most datasets the argmax baseline matches or approaches the performance of billion-parameter LLMs.