# Metadata-based Data Exploration with Retrieval-Augmented Generation for Large Language Models

arXiv: https://arxiv.org/abs/2410.04231 · HTML: https://mlsanity.com/paper/2410.04231/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 5 Oct 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TABBIE: Pretrained Representations of Tabular Data](https://mlsanity.com/paper/2105.02584/) (2021)
- [Table2Vec: Neural Word and Entity Embeddings for Table Population and Retrieval](https://mlsanity.com/paper/1906.00041/) (2019)
- [Dataset search: a survey](https://mlsanity.com/paper/1901.00735/) (2019)
- [Extraction of Research Objectives, Machine Learning Model Names, and Dataset Names from Academic Papers and Analysis of Their Interrelationships Using LLM and Network Analysis](https://mlsanity.com/paper/2408.12097/) (2024)
- [Data Combination for Problem-solving: A Case of an Open Data Exchange Platform](https://mlsanity.com/paper/2012.11746/) (2020)

## Built on by (most cited first, 4 of 4)

- [Making Sense of Data in the Wild: Data Analysis Automation at Scale](https://mlsanity.com/paper/2502.15718/) (2025)
- [Flexible metadata harvesting for ecology using large language models](https://mlsanity.com/paper/2508.20115/) (2025)
- [Position: Multi-Agent Algorithmic Care Systems Demand Contestability for Trustworthy AI](https://mlsanity.com/paper/2603.20595/) (2026)
- [PIPER: Content-Based Table Search via profiling and LLM-Generated Pseudoqueries](https://mlsanity.com/paper/2605.18199/) (2026)

## Abstract (arXiv)

Developing the capacity to effectively search for requisite datasets is an urgent requirement to assist data users in identifying relevant datasets considering the very limited available metadata. For this challenge, the utilization of third-party data is emerging as a valuable source for improvement. Our research introduces a new architecture for data exploration which employs a form of Retrieval-Augmented Generation (RAG) to enhance metadata-based data discovery. The system integrates large language models (LLMs) with external vector databases to identify semantic relationships among diverse types of datasets. The proposed framework offers a new method for evaluating semantic similarity among heterogeneous data sources and for improving data exploration. Our study includes experimental results on four critical tasks: 1) recommending similar datasets, 2) suggesting combinable datasets, 3) estimating tags, and 4) predicting variables. Our results demonstrate that RAG can enhance the selection of relevant datasets, particularly from different categories, when compared to conventional metadata approaches. However, performance varied across tasks and models, which confirms the significance of selecting appropriate techniques based on specific use cases. The findings suggest that this approach holds promise for addressing challenges in data exploration and discovery, although further refinement is necessary for estimation tasks.