# Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences

arXiv: https://arxiv.org/abs/2401.01641 · HTML: https://mlsanity.com/paper/2401.01641/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 3 Jan 2024.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning](https://mlsanity.com/paper/2110.04361/) (2021)
- [Selfie: Self-supervised Pretraining for Image Embedding](https://mlsanity.com/paper/1906.02940/) (2019)
- [CoLES: Contrastive Learning for Event Sequences with Self-Supervision](https://mlsanity.com/paper/2002.08232/) (2020)
- [LaundroGraph: Self-Supervised Graph Representation Learning for Anti-Money Laundering](https://mlsanity.com/paper/2210.14360/) (2022)
- [Navigating the Dynamics of Financial Embeddings over Time](https://mlsanity.com/paper/2007.00591/) (2020)

## Built on by (most cited first, 11 of 11)

- [Multimodal Banking Dataset: Understanding Client Needs through Event Sequences](https://mlsanity.com/paper/2409.17587/) (2024)
- [Evaluating Fairness in Transaction Fraud Models: Fairness Metrics, Bias Audits, and Challenges](https://mlsanity.com/paper/2409.04373/) (2024)
- [LATTE: Learning Aligned Transactions and Textual Embeddings for Bank Clients](https://mlsanity.com/paper/2508.10021/) (2025)
- [Open Banking Foundational Model: Learning Language Representations from Few Financial Transactions](https://mlsanity.com/paper/2511.12154/) (2025)
- [Enhancing Foundation Models in Transaction Understanding with LLM-based Sentence Embeddings](https://mlsanity.com/paper/2601.05271/) (2025)
- [TransactionGPT](https://mlsanity.com/paper/2511.08939/) (2025)
- [SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training](https://mlsanity.com/paper/2606.21356/) (2026)
- [A Foundation Model for Multimodal Event Sequences in Financial Applications](https://mlsanity.com/paper/2607.09955/) (2026)
- [MINT: A Universal Zero-Shot Predictor for Transaction Data](https://mlsanity.com/paper/2608.14198/) (2026)
- [Transfer Learning for Evolving Domains](https://mlsanity.com/paper/2609.13039/) (2026)
- [From "Who Is This User?" to "What Does This Purchase Mean?": A Deployed Pipeline for Semantic User Profiling at Bank Scale](https://mlsanity.com/paper/2609.19928/) (2026)

## Abstract (arXiv)

Machine learning models underpin many modern financial systems for use cases such as fraud detection and churn prediction. Most are based on supervised learning with hand-engineered features, which relies heavily on the availability of labelled data. Large self-supervised generative models have shown tremendous success in natural language processing and computer vision, yet so far they haven't been adapted to multivariate time series of financial transactions. In this paper, we present a generative pretraining method that can be used to obtain contextualised embeddings of financial transactions. Benchmarks on public datasets demonstrate that it outperforms state-of-the-art self-supervised methods on a range of downstream tasks. We additionally perform large-scale pretraining of an embedding model using a corpus of data from 180 issuing banks containing 5.1 billion transactions and apply it to the card fraud detection problem on hold-out datasets. The embedding model significantly improves value detection rate at high precision thresholds and transfers well to out-of-domain distributions.