# TabSketchFM: Sketch-based Tabular Representation Learning for Data Discovery over Data Lakes

arXiv: https://arxiv.org/abs/2407.01619 · HTML: https://mlsanity.com/paper/2407.01619/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 75 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [A Survey of Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2202.10936/) (2022)
- [GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing](https://mlsanity.com/paper/2009.13845/) (2020)
- [TABBIE: Pretrained Representations of Tabular Data](https://mlsanity.com/paper/2105.02584/) (2021)
- [TUTA: Tree-based Transformers for Generally Structured Table Pre-training](https://mlsanity.com/paper/2010.12537/) (2020)
- [Sherlock: A Deep Learning Approach to Semantic Data Type Detection](https://mlsanity.com/paper/1905.10688/) (2019)
- [GitTables: A Large-Scale Corpus of Relational Tables](https://mlsanity.com/paper/2106.07258/) (2021)
- [Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning](https://mlsanity.com/paper/2210.01922/) (2022)
- [Annotating Columns with Pre-trained Language Models](https://mlsanity.com/paper/2104.01785/) (2021)
- [Valentine: Evaluating Matching Techniques for Dataset Discovery](https://mlsanity.com/paper/2010.07386/) (2020)
- [SANTOS: Relationship-based Semantic Table Union Search](https://mlsanity.com/paper/2209.13589/) (2022)
- [Dataset Discovery in Data Lakes](https://mlsanity.com/paper/2011.10427/) (2020)
- [VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repository](https://mlsanity.com/paper/1905.04616/) (2019)
- [LSH Ensemble: Internet-Scale Domain Search](https://mlsanity.com/paper/1603.07410/) (2016)
- [Retrieving Complex Tables with Multi-Granular Graph Representation Learning](https://mlsanity.com/paper/2105.01736/) (2021)
- [CHORUS: Foundation Models for Unified Data Discovery and Exploration](https://mlsanity.com/paper/2306.09610/) (2023)
- [Efficient Joinable Table Discovery in Data Lakes: A High-Dimensional Similarity-Based Approach](https://mlsanity.com/paper/2010.13273/) (2020)
- [Web Table Extraction, Retrieval and Augmentation: A Survey](https://mlsanity.com/paper/2002.00207/) (2020)
- [DeepJoin: Joinable Table Discovery with Pre-trained Language Models](https://mlsanity.com/paper/2212.07588/) (2022)
- [WarpGate: A Semantic Join Discovery System for Cloud Data Warehouses](https://mlsanity.com/paper/2212.14155/) (2022)
- [LakeBench: Benchmarks for Data Discovery over Data Lakes](https://mlsanity.com/paper/2307.04217/) (2023)
- [Correlation Sketches for Approximate Join-Correlation Queries](https://mlsanity.com/paper/2104.03353/) (2021)
- [MATE: Multi-Attribute Table Extraction](https://mlsanity.com/paper/2110.00318/) (2021)

## Built on by (most cited first, 1 of 1)

- [MosaicJoin: Compact Semantic Sketches for Value-Level Join Discovery](https://mlsanity.com/paper/2607.21781/) (2026)

## Abstract (arXiv)

Enterprises have a growing need to identify relevant tables in data lakes; e.g. tables that are unionable, joinable, or subsets of each other. Tabular neural models can be helpful for such data discovery tasks. In this paper, we present TabSketchFM, a neural tabular model for data discovery over data lakes. First, we propose novel pre-training: a sketch-based approach to enhance the effectiveness of data discovery in neural tabular models. Second, we finetune the pretrained model for identifying unionable, joinable, and subset table pairs and show significant improvement over previous tabular neural models. Third, we present a detailed ablation study to highlight which sketches are crucial for which tasks. Fourth, we use these finetuned models to perform table search; i.e., given a query table, find other tables in a corpus that are unionable, joinable, or that are subsets of the query. Our results demonstrate significant improvements in F1 scores for search compared to state-of-the-art techniques. Finally, we show significant transfer across datasets and tasks establishing that our model can generalize across different tasks and over different data lakes.