# Transformers Meet Relational Databases

arXiv: https://arxiv.org/abs/2412.05218 · HTML: https://mlsanity.com/paper/2412.05218/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 73 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [Tracr: Compiled Transformers as a Laboratory for Interpretability](https://mlsanity.com/paper/2301.05062/) (2023)
- [RelBench: A Benchmark for Deep Learning on Relational Databases](https://mlsanity.com/paper/2407.20060/) (2024)
- [Supervised Learning on Relational Databases with Graph Neural Networks](https://mlsanity.com/paper/2002.02046/) (2020)
- [Turning 30: New Ideas in Inductive Logic Programming](https://mlsanity.com/paper/2002.11002/) (2020)
- [Trompt: Towards a Better Deep Neural Network for Tabular Data](https://mlsanity.com/paper/2305.18446/) (2023)
- [ExcelFormer: A neural network surpassing GBDTs on tabular data](https://mlsanity.com/paper/2301.02819/) (2023)
- [PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learning](https://mlsanity.com/paper/2404.00776/) (2024)
- [GFS: Graph-based Feature Synthesis for Prediction over Relational Databases](https://mlsanity.com/paper/2312.02037/) (2023)
- [Towards Foundation Models for Relational Databases [Vision Paper]](https://mlsanity.com/paper/2305.15321/) (2023)

## Built on by (most cited first, 9 of 9)

- [RelBench v2: A Large-Scale Benchmark and Repository for Relational Data](https://mlsanity.com/paper/2602.12606/) (2026)
- [No Need to Train Your RDB Foundation Model](https://mlsanity.com/paper/2602.13697/) (2026)
- [End-to-End Compression for Tabular Foundation Models](https://mlsanity.com/paper/2602.05649/) (2026)
- [A Fair Benchmarking of Deep Relational Database Learning Models](https://mlsanity.com/paper/2607.03659/) (2026)
- [Universal Encoders for Modular Relational Deep Learning](https://mlsanity.com/paper/2606.21434/) (2026)
- [Incremental Evaluation and Training in Relational Deep Learning](https://mlsanity.com/paper/2608.13023/) (2026)
- [Relational Task Generation Language: A Declarative Specification Framework for Relational Deep Learning](https://mlsanity.com/paper/2609.01292/) (2026)
- [STEER: Reducing Inference Cost in Relational Foundation Models through Semantically Informed Sampling](https://mlsanity.com/paper/2610.00907/) (2026)
- [RelICL: Training-free Relational Learning with Tabular Foundation Models](https://mlsanity.com/paper/2610.01725/) (2026)

## Abstract (arXiv)

Transformer models have continuously expanded into all machine learning domains convertible to the underlying sequence-to-sequence representation, including tabular data. However, while ubiquitous, this representation restricts their extension to the more general case of relational databases. In this paper, we introduce a modular neural message-passing scheme that closely adheres to the formal relational model, enabling direct end-to-end learning of tabular Transformers from database storage systems. We address the challenges of appropriate learning data representation and loading, which are critical in the database setting, and compare our approach against a number of representative models from various related fields across a significantly wide range of datasets. Our results demonstrate a superior performance of this newly proposed class of neural architectures.