# Large Language Models are Good Relational Learners

arXiv: https://arxiv.org/abs/2506.05725 · HTML: https://mlsanity.com/paper/2506.05725/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 75 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Builds on (most cited first, 30 of 48)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [A Comprehensive Survey on Graph Anomaly Detection with Deep Learning](https://mlsanity.com/paper/2106.07178/) (2021)
- [Holographic Embeddings of Knowledge Graphs](https://mlsanity.com/paper/1510.04935/) (2015)
- [Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning](https://mlsanity.com/paper/2301.13808/) (2023)
- [What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning](https://mlsanity.com/paper/2305.09731/) (2023)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Large Language Models are few(1)-shot Table Reasoners](https://mlsanity.com/paper/2210.06710/) (2022)
- [TABBIE: Pretrained Representations of Tabular Data](https://mlsanity.com/paper/2105.02584/) (2021)
- [TUTA: Tree-based Transformers for Generally Structured Table Pre-training](https://mlsanity.com/paper/2010.12537/) (2020)
- [LocAgent: Graph-Guided LLM Agents for Code Localization](https://mlsanity.com/paper/2503.09089/) (2025)

## Built on by (most cited first, 24 of 24)

- [Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data](https://mlsanity.com/paper/2510.06377/) (2025)
- [Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2509.21882/) (2025)
- [DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search](https://mlsanity.com/paper/2509.25454/) (2025)
- [No Need to Train Your RDB Foundation Model](https://mlsanity.com/paper/2602.13697/) (2026)
- [RDBLearn: Simple In-Context Prediction Over Relational Databases](https://mlsanity.com/paper/2602.18495/) (2026)
- [Towards Anomaly Detection on Relational Data](https://mlsanity.com/paper/2606.18621/) (2026)
- [OpenRFM: Dissecting Relational In-Context Learning](https://mlsanity.com/paper/2606.04320/) (2026)
- [ReFuGe: Feature Generation for Prediction Tasks on Relational Databases with LLM Agents](https://mlsanity.com/paper/2601.17735/) (2026)
- [Relatron: Automating Relational Machine Learning over Relational Databases](https://mlsanity.com/paper/2602.22552/) (2026)
- [RelAgent: LLM Agents as Data Scientists for Relational Learning](https://mlsanity.com/paper/2605.07840/) (2026)
- [Self-Exploring Language Models for Explainable Link Forecasting on Temporal Graphs via Reinforcement Learning](https://mlsanity.com/paper/2509.00975/) (2025)
- [From Pixels to Policies: Reinforcing Spatial Reasoning in Language Models for Content-Aware Layout Design](https://mlsanity.com/paper/2602.13912/) (2026)
- [Gaussian Relational Graph Transformer](https://mlsanity.com/paper/2605.15575/) (2026)
- [Relational Structural Causal Models](https://mlsanity.com/paper/2606.14892/) (2026)
- [Beyond Accuracy: Measuring Logical Compliance of Predictive Models](https://mlsanity.com/paper/2606.20208/) (2026)
- [Universal Encoders for Modular Relational Deep Learning](https://mlsanity.com/paper/2606.21434/) (2026)
- [InvWeaver: Deductive Feedback for Invariant Synthesis in Interacting-Loop Programs](https://mlsanity.com/paper/2607.05478/) (2026)
- [Incremental Evaluation and Training in Relational Deep Learning](https://mlsanity.com/paper/2608.13023/) (2026)
- [Identifiability and Order-Dimension Limits of In-Context Learning on Partial Orders](https://mlsanity.com/paper/2608.14004/) (2026)
- [Can LLMs Use Relational Transformer Embeddings?](https://mlsanity.com/paper/2609.00457/) (2026)
- [REFINE: LLM Refinement over Budgeted Text-Attributed Graphs for Personalized Medical Concept Representation](https://mlsanity.com/paper/2609.04415/) (2026)
- [CARAT: Do Materials LLMs Reason or Recite?](https://mlsanity.com/paper/2609.38340/) (2026)
- [STEER: Reducing Inference Cost in Relational Foundation Models through Semantically Informed Sampling](https://mlsanity.com/paper/2610.00907/) (2026)
- [RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions](https://mlsanity.com/paper/2610.17867/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities across various domains, yet their application to relational deep learning (RDL) remains underexplored. Existing approaches adapt LLMs by traversing relational links between entities in a database and converting the structured data into flat text documents. Still, this text-based serialization disregards critical relational structures, introduces redundancy, and often exceeds standard LLM context lengths. We introduce Rel-LLM, a novel architecture that utilizes a graph neural network (GNN)- based encoder to generate structured relational prompts for LLMs within a retrieval-augmented generation (RAG) framework. Unlike traditional text-based serialization approaches, our method preserves the inherent relational structure of databases while enabling LLMs to effectively process and reason over complex entity relationships. Specifically, the GNN encoder extracts a local subgraph around an entity to build feature representations that contain relevant entity relationships and temporal dependencies. These representations are transformed into structured prompts using a denormalization process, effectively allowing the LLM to reason over relational structures. Through extensive experiments, we demonstrate that Rel-LLM outperforms existing methods on key RDL tasks, offering a scalable and efficient approach to integrating LLMs with structured data sources. Code is available at https://github.com/smiles724/Rel-LLM.