# Dataset Discovery in Data Lakes

arXiv: https://arxiv.org/abs/2011.10427 · HTML: https://mlsanity.com/paper/2011.10427/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2020.

## Built on by (most cited first, 30 of 31)

- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Data-centric AI: Perspectives and Challenges](https://mlsanity.com/paper/2301.04819/) (2023)
- [Jellyfish: A Large Language Model for Data Preprocessing](https://mlsanity.com/paper/2312.01678/) (2023)
- [DeepJoin: Joinable Table Discovery with Pre-trained Language Models](https://mlsanity.com/paper/2212.07588/) (2022)
- [Tabular Data Augmentation for Machine Learning: Progress and Prospects of Embracing Generative AI](https://mlsanity.com/paper/2407.21523/) (2024)
- [Observatory: Characterizing Embeddings of Relational Tables](https://mlsanity.com/paper/2310.07736/) (2023)
- [Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes](https://mlsanity.com/paper/2104.04659/) (2021)
- [Cross Modal Data Discovery over Structured and Unstructured Data Lakes](https://mlsanity.com/paper/2306.00932/) (2023)
- [YOLOv8 for Defect Inspection of Hexagonal Directed Self-Assembly Patterns: A Data-Centric Approach](https://mlsanity.com/paper/2307.15516/) (2023)
- [Snoopy: Effective and Efficient Semantic Join Discovery via Proxy Columns](https://mlsanity.com/paper/2502.16813/) (2025)
- [Retrieval-Based Transformer for Table Augmentation](https://mlsanity.com/paper/2306.11843/) (2023)
- [KGLiDS: A Platform for Semantic Abstraction, Linking, and Automation of Data Science](https://mlsanity.com/paper/2303.02204/) (2023)
- [Adversarial Learning for Feature Shift Detection and Correction](https://mlsanity.com/paper/2312.04546/) (2023)
- [Model Lakes](https://mlsanity.com/paper/2403.02327/) (2024)
- [DALL-M: Context-Aware Clinical Data Augmentation with LLMs](https://mlsanity.com/paper/2407.08227/) (2024)
- [HyperJoin: LLM-augmented Hypergraph Link Prediction for Joinable Table Discovery](https://mlsanity.com/paper/2601.01015/) (2026)
- [Generative Benchmark Creation for Table Union Search](https://mlsanity.com/paper/2308.03883/) (2023)
- [TabSketchFM: Sketch-based Tabular Representation Learning for Data Discovery over Data Lakes](https://mlsanity.com/paper/2407.01619/) (2024)
- [Tabular Embeddings for Tables with Bi-Dimensional Hierarchical Metadata and Nesting](https://mlsanity.com/paper/2502.15819/) (2025)
- [Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks](https://mlsanity.com/paper/2507.02819/) (2025)
- [Conceptual Schema Inference for Tabular Datasets using Large Language Models](https://mlsanity.com/paper/2509.04632/) (2025)
- [ModelTables: A Corpus of Tables about Models](https://mlsanity.com/paper/2512.16106/) (2025)
- [Nexus: Inferring Join Graphs from Metadata Alone via Iterative Low-Rank Matrix Completion](https://mlsanity.com/paper/2602.08186/) (2026)
- [PIPER: Content-Based Table Search via profiling and LLM-Generated Pseudoqueries](https://mlsanity.com/paper/2605.18199/) (2026)
- [SANA: What Matters for QA Agents over Massive Data Lakes?](https://mlsanity.com/paper/2606.13904/) (2026)
- [Alignment-Guided Largest Table Overlap Size Estimation](https://mlsanity.com/paper/2607.03049/) (2026)
- [Faithful or Findable? Evaluating LLM-Generated Metadata for RDF Dataset Search](https://mlsanity.com/paper/2607.05970/) (2026)
- [Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Models](https://mlsanity.com/paper/2607.19847/) (2026)
- [MosaicJoin: Compact Semantic Sketches for Value-Level Join Discovery](https://mlsanity.com/paper/2607.21781/) (2026)
- [Discovering Relationships in Data Lakes Using Large Language Models: An Industrial Case](https://mlsanity.com/paper/2608.26750/) (2026)

## Abstract (arXiv)

Data analytics stands to benefit from the increasing availability of datasets that are held without their conceptual relationships being explicitly known. When collected, these datasets form a data lake from which, by processes like data wrangling, specific target datasets can be constructed that enable value-adding analytics. Given the potential vastness of such data lakes, the issue arises of how to pull out of the lake those datasets that might contribute to wrangling out a given target. We refer to this as the problem of dataset discovery in data lakes and this paper contributes an effective and efficient solution to it. Our approach uses features of the values in a dataset to construct hash-based indexes that map those features into a uniform distance space. This makes it possible to define similarity distances between features and to take those distances as measurements of relatedness w.r.t. a target table. Given the latter (and exemplar tuples), our approach returns the most related tables in the lake. We provide a detailed description of the approach and report on empirical results for two forms of relatedness (unionability and joinability) comparing them with prior work, where pertinent, and showing significant improvements in all of precision, recall, target coverage, indexing and discovery times.