# A Foundation Model for Multimodal Event Sequences in Financial Applications

arXiv: https://arxiv.org/abs/2607.09955 · HTML: https://mlsanity.com/paper/2607.09955/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 10 Jul 2026.

## Builds on (most cited first, 23 of 23)

- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [PinnerFormer: Sequence Modeling for User Representation at Pinterest](https://mlsanity.com/paper/2205.04507/) (2022)
- [Making Pre-trained Language Models Great on Tabular Prediction](https://mlsanity.com/paper/2403.01841/) (2024)
- [TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest](https://mlsanity.com/paper/2306.00248/) (2023)
- [CoLES: Contrastive Learning for Event Sequences with Self-Supervision](https://mlsanity.com/paper/2002.08232/) (2020)
- [Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences](https://mlsanity.com/paper/2401.01641/) (2024)
- [Multimodal Banking Dataset: Understanding Client Needs through Event Sequences](https://mlsanity.com/paper/2409.17587/) (2024)
- [Scaling Recommender Transformers to One Billion Parameters](https://mlsanity.com/paper/2507.15994/) (2025)
- [FT-Transformer: Resilient and Reliable Transformer with End-to-End Fault Tolerant Attention](https://mlsanity.com/paper/2504.02211/) (2025)
- [DV365: Extremely Long User History Modeling at Instagram](https://mlsanity.com/paper/2506.00450/) (2025)
- [PinFM: Foundation Model for User Activity Sequences at a Billion-scale Visual Discovery Platform](https://mlsanity.com/paper/2507.12704/) (2025)
- [MLEM: Generative and Contrastive Learning as Distinct Modalities for Event Sequences](https://mlsanity.com/paper/2401.15935/) (2024)
- [Learning Transactions Representations for Information Management in Banks: Mastering Local, Global, and External Knowledge](https://mlsanity.com/paper/2404.02047/) (2024)
- [Detecting the Future: All-at-Once Event Sequence Forecasting with Horizon Matching](https://mlsanity.com/paper/2408.13131/) (2024)
- [Scalable Representation Learning for Multimodal Tabular Transactions](https://mlsanity.com/paper/2410.07851/) (2024)
- [Your Spending Needs Attention: Modeling Financial Habits with Transformers](https://mlsanity.com/paper/2507.23267/) (2025)
- [LLM4ES: Learning User Embeddings from Event Sequences via Large Language Models](https://mlsanity.com/paper/2508.05688/) (2025)
- [Encode Me If You Can: Learning Universal User Representations via Event Sequence Autoencoding](https://mlsanity.com/paper/2508.07748/) (2025)
- [HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens](https://mlsanity.com/paper/2508.01474/) (2025)
- [Open Banking Foundational Model: Learning Language Representations from Few Financial Transactions](https://mlsanity.com/paper/2511.12154/) (2025)
- [Blending Sequential Embeddings, Graphs, and Engineered Features: 4th Place Solution in RecSys Challenge 2025](https://mlsanity.com/paper/2508.06970/) (2025)

## Abstract (arXiv)

Predictive modeling is a core component of modern financial services, where a wide range of tasks are traditionally addressed using separate models trained on manually engineered tabular features. This task-specific approach limits reuse and makes it difficult to fully exploit heterogeneous data sources such as transaction histories and digital interaction signals. In this paper, we present an approach based on pretraining a foundation transformer model on multimodal sequences of user events. Events from multiple data sources are unified into a single chronological sequence, enabling early fusion of heterogeneous modalities and learning of general-purpose representations via a next-event prediction objective. These representations are combined with existing engineered user features, on top of which lightweight neural models are trained for multiple downstream tasks. The proposed system outperforms traditional task-specific models while reducing development overhead. The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics.