# Empowering Tabular Data Preparation with Language Models: Why and How?

arXiv: https://arxiv.org/abs/2508.01556 · HTML: https://mlsanity.com/paper/2508.01556/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 102 references that are in the map.
- First posted on arXiv 3 Aug 2025.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [Deep Entity Matching with Pre-Trained Language Models](https://mlsanity.com/paper/2004.00584/) (2020)
- [SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation](https://mlsanity.com/paper/2406.14991/) (2024)
- [MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes](https://mlsanity.com/paper/2412.19260/) (2024)
- [Open Domain Question Answering over Tables via Dense Retrieval](https://mlsanity.com/paper/2103.12011/) (2021)
- [Table-GPT: Table-tuned GPT for Diverse Table Tasks](https://mlsanity.com/paper/2310.09263/) (2023)
- [Large Language Model for Table Processing: A Survey](https://mlsanity.com/paper/2402.05121/) (2024)
- [Open-WikiTable: Dataset for Open Domain Question Answering with Complex Reasoning over Table](https://mlsanity.com/paper/2305.07288/) (2023)
- [Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning](https://mlsanity.com/paper/2210.01922/) (2022)
- [Valentine: Evaluating Matching Techniques for Dataset Discovery](https://mlsanity.com/paper/2010.07386/) (2020)
- [Large Language Models as Data Preprocessors](https://mlsanity.com/paper/2308.16361/) (2023)
- [Match, Compare, or Select? An Investigation of Large Language Models for Entity Matching](https://mlsanity.com/paper/2405.16884/) (2024)
- [LSH Ensemble: Internet-Scale Domain Search](https://mlsanity.com/paper/1603.07410/) (2016)
- [Retrieving Complex Tables with Multi-Granular Graph Representation Learning](https://mlsanity.com/paper/2105.01736/) (2021)
- [AlphaClean: Automatic Generation of Data Cleaning Pipelines](https://mlsanity.com/paper/1904.11827/) (2019)
- [(Almost) All of Entity Resolution](https://mlsanity.com/paper/2008.04443/) (2020)
- [Table Search Using a Deep Contextualized Language Model](https://mlsanity.com/paper/2005.09207/) (2020)

## Built on by (most cited first, 4 of 4)

- [Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs](https://mlsanity.com/paper/2601.17058/) (2026)
- [PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?](https://mlsanity.com/paper/2605.08687/) (2026)
- [Interpretable Column Annotation with LLM-Symbolized Decision Process Materialization](https://mlsanity.com/paper/2607.25228/) (2026)
- [Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go?](https://mlsanity.com/paper/2609.07629/) (2026)

## Abstract (arXiv)

Data preparation is a critical step in enhancing the usability of tabular data and thus boosts downstream data-driven tasks. Traditional methods often face challenges in capturing the intricate relationships within tables and adapting to the tasks involved. Recent advances in Language Models (LMs), especially in Large Language Models (LLMs), offer new opportunities to automate and support tabular data preparation. However, why LMs suit tabular data preparation (i.e., how their capabilities match task demands) and how to use them effectively across phases still remain to be systematically explored. In this survey, we systematically analyze the role of LMs in enhancing tabular data preparation processes, focusing on four core phases: data acquisition, integration, cleaning, and transformation. For each phase, we present an integrated analysis of how LMs can be combined with other components for different preparation tasks, highlight key advancements, and outline prospective pipelines.