# Something's Fishy In The Data Lake: A Critical Re-evaluation of Table Union Search Benchmarks

arXiv: https://arxiv.org/abs/2505.21329 · HTML: https://mlsanity.com/paper/2505.21329/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 52 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [GitTables: A Large-Scale Corpus of Relational Tables](https://mlsanity.com/paper/2106.07258/) (2021)
- [Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning](https://mlsanity.com/paper/2210.01922/) (2022)
- [HYTREL: Hypergraph-enhanced Tabular Data Representation Learning](https://mlsanity.com/paper/2307.08623/) (2023)
- [SANTOS: Relationship-based Semantic Table Union Search](https://mlsanity.com/paper/2209.13589/) (2022)
- [LSH Ensemble: Internet-Scale Domain Search](https://mlsanity.com/paper/1603.07410/) (2016)
- [Auctus: A Dataset Search Engine for Data Augmentation](https://mlsanity.com/paper/2102.05716/) (2021)
- [Pylon: Semantic Table Union Search in Data Lakes](https://mlsanity.com/paper/2301.04901/) (2023)
- [Unveiling Challenges for LLMs in Enterprise Data Engineering](https://mlsanity.com/paper/2504.10950/) (2025)
- [How well do LLMs reason over tabular data, really?](https://mlsanity.com/paper/2505.07453/) (2025)
- [Retrieve, Merge, Predict: Augmenting Tables with Data Lakes](https://mlsanity.com/paper/2402.06282/) (2024)
- [LakeVisage: Towards Scalable, Flexible and Interactive Visualization Recommendation for Data Discovery over Data Lakes](https://mlsanity.com/paper/2504.02150/) (2025)
- [Metadata-driven Table Union Search: Leveraging Semantics for Restricted Access Data Integration](https://mlsanity.com/paper/2502.20945/) (2025)
- [NLCTables: A Dataset for Marrying Natural Language Conditions with Table Discovery](https://mlsanity.com/paper/2504.15849/) (2025)

## Built on by (most cited first, 1 of 1)

- [Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks](https://mlsanity.com/paper/2604.21696/) (2026)

## Abstract (arXiv)

Recent table representation learning and data discovery methods tackle table union search (TUS) within data lakes, which involves identifying tables that can be unioned with a given query table to enrich its content. These methods are commonly evaluated using benchmarks that aim to assess semantic understanding in real-world TUS tasks. However, our analysis of prominent TUS benchmarks reveals several limitations that allow simple baselines to perform surprisingly well, often outperforming more sophisticated approaches. This suggests that current benchmark scores are heavily influenced by dataset-specific characteristics and fail to effectively isolate the gains from semantic understanding. To address this, we propose essential criteria for future benchmarks to enable a more realistic and reliable evaluation of progress in semantic table union search.