# AutoDDG: Automated Dataset Description Generation using Large Language Models

arXiv: https://arxiv.org/abs/2502.01050 · HTML: https://mlsanity.com/paper/2502.01050/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2025.

## Built on by (most cited first, 8 of 8)

- [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025)
- [Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs](https://mlsanity.com/paper/2601.17058/) (2026)
- [A Reference Architecture for Agentic Hybrid Retrieval in Dataset Search](https://mlsanity.com/paper/2604.16394/) (2026)
- [LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake](https://mlsanity.com/paper/2606.10460/) (2026)
- [PIPER: Content-Based Table Search via profiling and LLM-Generated Pseudoqueries](https://mlsanity.com/paper/2605.18199/) (2026)
- [SANA: What Matters for QA Agents over Massive Data Lakes?](https://mlsanity.com/paper/2606.13904/) (2026)
- [Faithful or Findable? Evaluating LLM-Generated Metadata for RDF Dataset Search](https://mlsanity.com/paper/2607.05970/) (2026)
- [MosaicJoin: Compact Semantic Sketches for Value-Level Join Discovery](https://mlsanity.com/paper/2607.21781/) (2026)

## Abstract (arXiv)

The proliferation of datasets across open data portals and enterprise data lakes presents an opportunity for deriving data-driven insights. Widely-used dataset search systems rely on keyword search over dataset metadata, including descriptions, to support discovery. Therefore, when these descriptions are incomplete, missing, or inconsistent with dataset contents, findability is severely compromised. To improve findability, we introduce AutoDDG, a framework that automatically generates descriptions of tabular data. By adopting a data-driven approach to summarize dataset contents and leveraging large language models (LLMs) to enrich summaries with semantic information and produce human-readable text, AutoDDG derives descriptions that are comprehensive, accurate, readable, and concise. A critical challenge in this problem is evaluating the effectiveness of description generation methods and assessing the quality of the generated descriptions. We propose a comprehensive evaluation methodology that combines retrieval, reference-based, and reference-free assessment, with human validation. Our experimental results using new benchmarks demonstrate that AutoDDG generates high-quality, accurate descriptions at scale, significantly improving dataset retrieval performance across diverse use cases. AutoDDG is publicly available at https://github.com/VIDA-NYU/AutoDDG.