# RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation

arXiv: https://arxiv.org/abs/2012.02469 · HTML: https://mlsanity.com/paper/2012.02469/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2020.

## Built on by (most cited first, 27 of 27)

- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Generative Table Pre-training Empowers Models for Tabular Prediction](https://mlsanity.com/paper/2305.09696/) (2023)
- [Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks](https://mlsanity.com/paper/2201.09745/) (2022)
- [Annotating Columns with Pre-trained Language Models](https://mlsanity.com/paper/2104.01785/) (2021)
- [TabLib: A Dataset of 627M Tables with Context](https://mlsanity.com/paper/2310.07875/) (2023)
- [FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining](https://mlsanity.com/paper/2109.07323/) (2021)
- [RAG over Tables: Hierarchical Memory Index, Multi-Stage Retrieval, and Benchmarking](https://mlsanity.com/paper/2504.01346/) (2025)
- [VerifAI: Verified Generative AI](https://mlsanity.com/paper/2307.02796/) (2023)
- [DeepJoin: Joinable Table Discovery with Pre-trained Language Models](https://mlsanity.com/paper/2212.07588/) (2022)
- [Observatory: Characterizing Embeddings of Relational Tables](https://mlsanity.com/paper/2310.07736/) (2023)
- [E2ETune: End-to-End Knob Tuning via Fine-tuned Generative Language Model](https://mlsanity.com/paper/2404.11581/) (2024)
- [Generating Realistic Tabular Data with Large Language Models](https://mlsanity.com/paper/2410.21717/) (2024)
- [Towards Foundation Models for Relational Databases [Vision Paper]](https://mlsanity.com/paper/2305.15321/) (2023)
- [LLM-Enhanced Data Management](https://mlsanity.com/paper/2402.02643/) (2024)
- [AIGT: AI Generative Table Based on Prompt](https://mlsanity.com/paper/2412.18111/) (2024)
- [CodexDB: Generating Code for Processing SQL Queries using GPT-3 Codex](https://mlsanity.com/paper/2204.08941/) (2022)
- [TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoning](https://mlsanity.com/paper/2510.06217/) (2025)
- [Large Language Models for Imbalanced Classification: Diversity makes the difference](https://mlsanity.com/paper/2510.09783/) (2025)
- [S2abEL: A Dataset for Entity Linking from Scientific Tables](https://mlsanity.com/paper/2305.00366/) (2023)
- [TabulaX: Leveraging Large Language Models for Multi-Class Table Transformations](https://mlsanity.com/paper/2411.17110/) (2024)
- [Can Deep Neural Networks Predict Data Correlations from Column Names?](https://mlsanity.com/paper/2107.04553/) (2021)
- [Making Table Understanding Work in Practice](https://mlsanity.com/paper/2109.05173/) (2021)
- [Beyond Text-to-SQL for IoT Defense: A Comprehensive Framework for Querying and Classifying IoT Threats](https://mlsanity.com/paper/2406.17574/) (2024)
- [Table Transformers for Imputing Textual Attributes](https://mlsanity.com/paper/2408.02128/) (2024)
- [RoTaR: Efficient Row-Based Table Representation Learning via Teacher-Student Training](https://mlsanity.com/paper/2306.11696/) (2023)
- [An Automatic Prompt Generation System for Tabular Data Tasks](https://mlsanity.com/paper/2405.05618/) (2024)
- [Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks](https://mlsanity.com/paper/2605.16085/) (2026)

## Abstract (arXiv)

Can AI help automate human-easy but computer-hard data preparation tasks that burden data scientists, practitioners, and crowd workers? We answer this question by presenting RPT, a denoising auto-encoder for tuple-to-X models (X could be tuple, token, label, JSON, and so on). RPT is pre-trained for a tuple-to-tuple model by corrupting the input tuple and then learning a model to reconstruct the original tuple. It adopts a Transformer-based neural translation architecture that consists of a bidirectional encoder (similar to BERT) and a left-to-right autoregressive decoder (similar to GPT), leading to a generalization of both BERT and GPT. The pre-trained RPT can already support several common data preparation tasks such as data cleaning, auto-completion and schema matching. Better still, RPT can be fine-tuned on a wide range of data preparation tasks, such as value normalization, data transformation, data annotation, etc. To complement RPT, we also discuss several appealing techniques such as collaborative training and few-shot learning for entity resolution, and few-shot learning and NLP question-answering for information extraction. In addition, we identify a series of research opportunities to advance the field of data preparation.