mlsanityData to 14 Oct 2026

SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training

arXiv 2606.21356 ↗First posted 19 Jun 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Many machine learning applications rely on heterogeneous event streams to make predictions, either causally as events arrive or bidirectionally over complete sequences. We propose SOHET (Sequence Of Heterogeneous Events Transformer), a hierarchical architecture combining event-type-specific tabular encoders with temporal and type embeddings, processed by a causal or bidirectional transformer. We introduce three self-supervised pre-training objectives for the causal setting. On a proprietary large-scale real-world Booking.com fraud detection task with 17 event types, SOHET outperforms FlexTPP, NAPPT, and CIPPT by 5.8%. Pre-training yields an additional 2.6% gain and 2.4% faster convergence. On the EBES benchmark, bidirectional SOHET matches or exceeds the published best on 6 out of 8 tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

26 of 32 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  6. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  7. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  8. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  9. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  10. 2021Multi-Time Attention Networks for Irregularly Sampled Time Seriescited by 238
  11. 2020Transformer Hawkes Processcited by 235
  12. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
14 more
  1. 2019Behavior Sequence Transformer for E-commerce Recommendation in Alibabacited by 130
  2. 2020Tabular Transformers for Modeling Multivariate Time Seriescited by 77
  3. 2021CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction taskscited by 76
  4. 2023Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Eventscited by 46
  5. 2023Improving Position Encoding of Transformers for Multivariate Time Series Classificationcited by 45
  6. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 38
  7. 2023TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterestcited by 35
  8. 2020CoLES: Contrastive Learning for Event Sequences with Self-Supervisioncited by 32
  9. 2023FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Datacited by 14
  10. 2023One Transformer for All Time Series: Representing and Training with Time-Dependent Heterogeneous Tabular Datacited by 13
  11. 2024Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequencescited by 11
  12. 2024EBES: Easy Benchmarking for Event Sequencescited by 7
  13. 2024MLEM: Generative and Contrastive Learning as Distinct Modalities for Event Sequencescited by 6
  14. 2025PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modelingcited by 4

Built on by

Not cited yet.