# A Case for Business Process-Specific Foundation Models

arXiv: https://arxiv.org/abs/2210.14739 · HTML: https://mlsanity.com/paper/2210.14739/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 53 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts](https://mlsanity.com/paper/2111.02358/) (2021)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [Predictive Business Process Monitoring with LSTM Neural Networks](https://mlsanity.com/paper/1612.02130/) (2016)
- [LSTM Networks for Data-Aware Remaining Time Prediction of Business Process Instances](https://mlsanity.com/paper/1711.03822/) (2017)
- [Detecting sudden and gradual drifts in business processes from execution traces](https://mlsanity.com/paper/2005.04016/) (2020)
- [A Conversational Digital Assistant for Intelligent Process Automation](https://mlsanity.com/paper/2007.13256/) (2020)
- [Conformance checking: A state-of-the-art literature review](https://mlsanity.com/paper/2007.10903/) (2020)
- [Risks of AI Foundation Models in Education](https://mlsanity.com/paper/2110.10024/) (2021)
- [D3BA: A Tool for Optimizing Business Processes Using Non-Deterministic Planning](https://mlsanity.com/paper/2001.02619/) (2020)
- [Deep Reinforcement Learning for Resource Allocation in Business Processes](https://mlsanity.com/paper/2104.00541/) (2021)

## Built on by (most cited first, 5 of 5)

- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [Automating the Enterprise with Foundation Models](https://mlsanity.com/paper/2405.03710/) (2024)
- [Synthesis of Mathematical programs from Natural Language Specifications](https://mlsanity.com/paper/2304.03287/) (2023)
- [How well can a large language model explain business processes as perceived by users?](https://mlsanity.com/paper/2401.12846/) (2024)
- [From Source to Target: Leveraging Transfer Learning for Predictive Process Monitoring in Organizations](https://mlsanity.com/paper/2508.08061/) (2025)

## Abstract (arXiv)

The inception of large language models has helped advance state-of-the-art performance on numerous natural language tasks. This has also opened the door for the development of foundation models for other domains and data modalities such as images, code, and music. In this paper, we argue that business process data representations have unique characteristics that warrant the development of a new class of foundation models to handle tasks like process mining, optimization, and decision making. These models should also tackle the unique challenges of applying AI to business processes which include data scarcity, multi-modal representations, domain specific terminology, and privacy concerns.