# Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data

arXiv: https://arxiv.org/abs/2510.06377 · HTML: https://mlsanity.com/paper/2510.06377/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 48 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Recipe for a General, Powerful, Scalable Graph Transformer](https://mlsanity.com/paper/2205.12454/) (2022)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers](https://mlsanity.com/paper/2012.15828/) (2020)
- [Flex Attention: A Programming Model for Generating Optimized Attention Kernels](https://mlsanity.com/paper/2412.05496/) (2024)
- [Sign and Basis Invariant Networks for Spectral Graph Representation Learning](https://mlsanity.com/paper/2202.13013/) (2022)
- [Talk like a Graph: Encoding Graphs for Large Language Models](https://mlsanity.com/paper/2310.04560/) (2023)
- [Let Your Graph Do the Talking: Encoding Structured Data for LLMs](https://mlsanity.com/paper/2402.05862/) (2024)
- [PRODIGY: Enabling In-context Learning Over Graphs](https://mlsanity.com/paper/2305.12600/) (2023)
- [RelBench: A Benchmark for Deep Learning on Relational Databases](https://mlsanity.com/paper/2407.20060/) (2024)
- [Towards Foundation Models for Knowledge Graph Reasoning](https://mlsanity.com/paper/2310.04562/) (2023)
- [CARTE: Pretraining and Transfer for Tabular Learning](https://mlsanity.com/paper/2402.16785/) (2024)
- [RelGNN: Composite Message Passing for Relational Deep Learning](https://mlsanity.com/paper/2502.06784/) (2025)
- [Griffin: Towards a Graph-Centric Relational Database Foundation Model](https://mlsanity.com/paper/2505.05568/) (2025)
- [Towards Foundational Models for Molecular Learning on Large-Scale Multi-Task Datasets](https://mlsanity.com/paper/2310.04292/) (2023)
- [Fully-inductive Node Classification on Arbitrary Graphs](https://mlsanity.com/paper/2405.20445/) (2024)
- [Turning Tabular Foundation Models into Graph Foundation Models](https://mlsanity.com/paper/2508.20906/) (2025)
- [On the Stability of Expressive Positional Encodings for Graphs](https://mlsanity.com/paper/2310.02579/) (2023)
- [Learning Efficient Positional Encodings with Graph Neural Networks](https://mlsanity.com/paper/2502.01122/) (2025)
- [Large Language Models are Good Relational Learners](https://mlsanity.com/paper/2506.05725/) (2025)

## Built on by (most cited first, 30 of 39)

- [RelBench v2: A Large-Scale Benchmark and Repository for Relational Data](https://mlsanity.com/paper/2602.12606/) (2026)
- [PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models](https://mlsanity.com/paper/2602.04029/) (2026)
- [KumoRFM-2: Scaling Foundation Models for Relational Learning](https://mlsanity.com/paper/2604.12596/) (2026)
- [TabPFN-3: Technical Report](https://mlsanity.com/paper/2605.13986/) (2026)
- [Relational In-Context Learning via Synthetic Pre-training with Structural Prior](https://mlsanity.com/paper/2603.03805/) (2026)
- [No Need to Train Your RDB Foundation Model](https://mlsanity.com/paper/2602.13697/) (2026)
- [Predictive Query Language: A Domain-Specific Language for Predictive Modeling on Relational Databases](https://mlsanity.com/paper/2602.09572/) (2026)
- [RDBLearn: Simple In-Context Prediction Over Relational Databases](https://mlsanity.com/paper/2602.18495/) (2026)
- [Towards Anomaly Detection on Relational Data](https://mlsanity.com/paper/2606.18621/) (2026)
- [OpenRFM: Dissecting Relational In-Context Learning](https://mlsanity.com/paper/2606.04320/) (2026)
- [ReFuGe: Feature Generation for Prediction Tasks on Relational Databases with LLM Agents](https://mlsanity.com/paper/2601.17735/) (2026)
- [Advancing Open and Reproducible Relational Learning: RelArena-α, TabPFN-Rel and RPI](https://mlsanity.com/paper/2608.16319/) (2026)
- [Task Scarcity and Label Leakage in Relational Transfer Learning](https://mlsanity.com/paper/2603.29914/) (2026)
- [RelAgent: LLM Agents as Data Scientists for Relational Learning](https://mlsanity.com/paper/2605.07840/) (2026)
- [Rel-MOSS: Towards Imbalanced Relational Deep Learning on Relational Databases](https://mlsanity.com/paper/2603.07916/) (2026)
- [RelPrism: A Multi-Faceted Pre-training Framework with Self-Generated Tasks for Relational Databases](https://mlsanity.com/paper/2605.23241/) (2026)
- [A Fair Benchmarking of Deep Relational Database Learning Models](https://mlsanity.com/paper/2607.03659/) (2026)
- [TransactionGPT](https://mlsanity.com/paper/2511.08939/) (2025)
- [Gaussian Relational Graph Transformer](https://mlsanity.com/paper/2605.15575/) (2026)
- [RelGT-AC: A Relational Graph Transformer for Autocomplete Tasks in Relational Databases](https://mlsanity.com/paper/2606.03040/) (2026)
- [Relational Structural Causal Models](https://mlsanity.com/paper/2606.14892/) (2026)
- [Universal Encoders for Modular Relational Deep Learning](https://mlsanity.com/paper/2606.21434/) (2026)
- [FlexTab: A Flexible Encoder-Decoder Architecture for In-Context Learning Across Diverse Tabular Tasks](https://mlsanity.com/paper/2606.30336/) (2026)
- [InvWeaver: Deductive Feedback for Invariant Synthesis in Interacting-Loop Programs](https://mlsanity.com/paper/2607.05478/) (2026)
- [Curriculum Matters: Data-Efficient Relational PFN Pretraining with Synthetic Data](https://mlsanity.com/paper/2607.29120/) (2026)
- [Incremental Evaluation and Training in Relational Deep Learning](https://mlsanity.com/paper/2608.13023/) (2026)
- [MetaSieve: Faster Relational Deep Learning through SQL-Based Metapath Selection](https://mlsanity.com/paper/2608.25903/) (2026)
- [Can LLMs Use Relational Transformer Embeddings?](https://mlsanity.com/paper/2609.00457/) (2026)
- [Context Window Failures in Relational Foundation Models](https://mlsanity.com/paper/2609.00460/) (2026)
- [Relational Task Generation Language: A Declarative Specification Framework for Relational Deep Learning](https://mlsanity.com/paper/2609.01292/) (2026)

## Abstract (arXiv)

Pretrained transformers readily adapt to new sequence modeling tasks via zero-shot prompting, but relational domains still lack architectures that transfer across datasets and tasks. The core challenge is the diversity of relational data, with varying heterogeneous schemas, graph structures and functional dependencies. In this paper, we present the Relational Transformer (RT) architecture, which can be pretrained on diverse relational databases and directly applied to unseen datasets and tasks without task- or dataset-specific fine-tuning, or retrieval of in-context examples. RT (i) incorporates task specification via task table prompting, (ii) tokenizes cells with table/column metadata, (iii) is pretrained via masked token prediction, and (iv) utilizes a novel Relational Attention mechanism over columns, rows, and primary-foreign key links. Pretrained on RelBench datasets spanning tasks such as churn and sales forecasting, RT attains strong zero-shot performance, averaging 93% of fully supervised AUROC on binary classification tasks with a single forward pass of a 22M parameter model, as opposed to 84% for a 27B LLM. Fine-tuning yields state-of-the-art results with high sample efficiency. Our experimental analyses show that RT's zero-shot transfer leverages task context, relational attention patterns and schema semantics. Overall, RT provides a practical path toward foundation models for relational data. Code, models, data: https://github.com/snap-stanford/relational-transformer.