# Task-Agnostic Contrastive Pretraining for Relational Deep Learning

arXiv: https://arxiv.org/abs/2506.22530 · HTML: https://mlsanity.com/paper/2506.22530/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [Temporal Graph Networks for Deep Learning on Dynamic Graphs](https://mlsanity.com/paper/2006.10637/) (2020)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [RelBench: A Benchmark for Deep Learning on Relational Databases](https://mlsanity.com/paper/2407.20060/) (2024)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [Supervised Learning on Relational Databases with Graph Neural Networks](https://mlsanity.com/paper/2002.02046/) (2020)
- [RelGNN: Composite Message Passing for Relational Deep Learning](https://mlsanity.com/paper/2502.06784/) (2025)
- [Turning 30: New Ideas in Inductive Logic Programming](https://mlsanity.com/paper/2002.11002/) (2020)
- [Trompt: Towards a Better Deep Neural Network for Tabular Data](https://mlsanity.com/paper/2305.18446/) (2023)
- [Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations](https://mlsanity.com/paper/2210.08345/) (2022)
- [GFS: Graph-based Feature Synthesis for Prediction over Relational Databases](https://mlsanity.com/paper/2312.02037/) (2023)

## Built on by (most cited first, 4 of 4)

- [RelGT-AC: A Relational Graph Transformer for Autocomplete Tasks in Relational Databases](https://mlsanity.com/paper/2606.03040/) (2026)
- [Universal Encoders for Modular Relational Deep Learning](https://mlsanity.com/paper/2606.21434/) (2026)
- [Incremental Evaluation and Training in Relational Deep Learning](https://mlsanity.com/paper/2608.13023/) (2026)
- [Relational Task Generation Language: A Declarative Specification Framework for Relational Deep Learning](https://mlsanity.com/paper/2609.01292/) (2026)

## Abstract (arXiv)

Relational Deep Learning (RDL) is an emerging paradigm that leverages Graph Neural Network principles to learn directly from relational databases by representing them as heterogeneous graphs. However, existing RDL models typically rely on task-specific supervised learning, requiring training separate models for each predictive task, which may hamper scalability and reuse. In this work, we propose a novel task-agnostic contrastive pretraining approach for RDL that enables database-wide representation learning. For that aim, we introduce three levels of contrastive objectives-row-level, link-level, and context-level-designed to capture the structural and semantic heterogeneity inherent to relational data. We implement the respective pretraining approach through a modular RDL architecture and an efficient sampling strategy tailored to the heterogeneous database setting. Our preliminary results on standard RDL benchmarks demonstrate that fine-tuning the pretrained models measurably outperforms training from scratch, validating the promise of the proposed methodology in learning transferable representations for relational data.