# Relational In-Context Learning via Synthetic Pre-training with Structural Prior arXiv: https://arxiv.org/abs/2603.03805 ยท HTML: https://mlsanity.com/paper/2603.03805/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 54 of its 77 references that are in the map. - First posted on arXiv 4 Mar 2026. ## Builds on (most cited first, 30 of 54) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020) - [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016) - [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016) - [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018) - [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021) - [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023) - [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022) - [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019) - [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020) - [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020) - [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021) - [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021) - [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025) - [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022) - [The Road Less Scheduled](https://mlsanity.com/paper/2405.15682/) (2024) - [Handling Missing Data with Graph Representation Learning](https://mlsanity.com/paper/2010.16418/) (2020) - [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022) - [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025) - [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024) - [ForecastPFN: Synthetically-Trained Zero-Shot Forecasting](https://mlsanity.com/paper/2311.01933/) (2023) - [RelBench: A Benchmark for Deep Learning on Relational Databases](https://mlsanity.com/paper/2407.20060/) (2024) - [Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data](https://mlsanity.com/paper/2407.04491/) (2024) - [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023) - [LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence](https://mlsanity.com/paper/2509.03505/) (2025) - [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025) - [CARTE: Pretraining and Transfer for Tabular Learning](https://mlsanity.com/paper/2402.16785/) (2024) - [Relational Deep Learning: Graph Representation Learning on Relational Databases](https://mlsanity.com/paper/2312.04615/) (2023) ## Built on by (most cited first, 14 of 14) - [KumoRFM-2: Scaling Foundation Models for Relational Learning](https://mlsanity.com/paper/2604.12596/) (2026) - [Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality](https://mlsanity.com/paper/2605.18971/) (2026) - [OpenRFM: Dissecting Relational In-Context Learning](https://mlsanity.com/paper/2606.04320/) (2026) - [KGPFN: Unlocking the Potential of Knowledge Graph Foundation Model via In-Context Learning](https://mlsanity.com/paper/2605.14907/) (2026) - [Statistically Indistinguishable, Operationally Distinct: A Formal Barrier for Tabular Foundation Models](https://mlsanity.com/paper/2606.29091/) (2026) - [Causal Foundation Models](https://mlsanity.com/paper/2609.03003/) (2026) - [RelGT-AC: A Relational Graph Transformer for Autocomplete Tasks in Relational Databases](https://mlsanity.com/paper/2606.03040/) (2026) - [InvWeaver: Deductive Feedback for Invariant Synthesis in Interacting-Loop Programs](https://mlsanity.com/paper/2607.05478/) (2026) - [PluRel-to-RDB-PFN: Schema-Guided Synthetic Relational Pretraining](https://mlsanity.com/paper/2607.29129/) (2026) - [Support-Set Target Leakage in Relational Foundation Models during In-Context Learning: Impact, Detection, and Mitigation](https://mlsanity.com/paper/2609.36384/) (2026) - [Support-Set Target Leakage in Relational Foundation Models during In-Context Learning: Model Dependence and Evaluation Reliability](https://mlsanity.com/paper/2609.36417/) (2026) - [When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior](https://mlsanity.com/paper/2610.03057/) (2026) - [Relational Learning as World Modeling](https://mlsanity.com/paper/2610.16763/) (2026) - [RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions](https://mlsanity.com/paper/2610.17867/) (2026) ## Abstract (arXiv) Relational Databases (RDBs) are the backbone of modern business, yet they lack foundation models comparable to those in text or vision. A key obstacle is that high-quality RDBs are private, scarce, and structurally heterogeneous, making internet-scale pre-training infeasible. To overcome this data scarcity, we introduce RDB-PFN, the first relational foundation model trained purely via synthetic data. Inspired by Prior-Data Fitted Networks (PFNs), where synthetic data generated from Structural Causal Models (SCMs) enables reasoning on single tables, we design a Relational Prior Generator to create an infinite stream of diverse RDBs from scratch. Pre-training on over 2 million synthetic single-table and relational tasks, RDB-PFN learns to adapt to any new database instantly via genuine in-context learning. Experiments show that RDB-PFN achieves strong few-shot performance on 19 real-world relational prediction tasks, outperforming state-of-the-art tabular foundation models evaluated on the same DFS-linearized inputs, while using a lightweight architecture and fast inference. The code is available at https://github.com/MuLabPKU/RDBPFN.