# Towards Foundation Models for Relational Databases [Vision Paper]

arXiv: https://arxiv.org/abs/2305.15321 · HTML: https://mlsanity.com/paper/2305.15321/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [Can Foundation Models Wrangle Your Data?](https://mlsanity.com/paper/2205.09911/) (2022)
- [Directed Acyclic Graph Neural Networks](https://mlsanity.com/paper/2101.07965/) (2021)
- [Supervised Learning on Relational Databases with Graph Neural Networks](https://mlsanity.com/paper/2002.02046/) (2020)
- [GitTables: A Large-Scale Corpus of Relational Tables](https://mlsanity.com/paper/2106.07258/) (2021)
- [Deep Unsupervised Cardinality Estimation](https://mlsanity.com/paper/1905.04278/) (2019)
- [RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation](https://mlsanity.com/paper/2012.02469/) (2020)
- [Efficient and effective training of language and graph neural network models](https://mlsanity.com/paper/2206.10781/) (2022)
- [Termite: A System for Tunneling Through Heterogeneous Data](https://mlsanity.com/paper/1903.05008/) (2019)

## Built on by (most cited first, 7 of 7)

- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data](https://mlsanity.com/paper/2510.06377/) (2025)
- [Graph Neural Networks for Tabular Data Learning: A Survey with Taxonomy and Directions](https://mlsanity.com/paper/2401.02143/) (2024)
- [Transformers Meet Relational Databases](https://mlsanity.com/paper/2412.05218/) (2024)
- [Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks](https://mlsanity.com/paper/2605.16085/) (2026)
- [LLMs+Graphs: Toward Graph-Native, Synergistic AI Systems](https://mlsanity.com/paper/2606.11560/) (2026)
- [RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions](https://mlsanity.com/paper/2610.17867/) (2026)

## Abstract (arXiv)

Tabular representation learning has recently gained a lot of attention. However, existing approaches only learn a representation from a single table, and thus ignore the potential to learn from the full structure of relational databases, including neighboring tables that can contain important information for a contextualized representation. Moreover, current models are significantly limited in scale, which prevents that they learn from large databases. In this paper, we thus introduce our vision of relational representation learning, that can not only learn from the full relational structure, but also can scale to larger database sizes that are commonly found in real-world. Moreover, we also discuss opportunities and challenges we see along the way to enable this vision and present initial very promising results. Overall, we argue that this direction can lead to foundation models for relational databases that are today only available for text and images.