# Navigating the Data Lake with Datamaran: Automatically Extracting Structure from Log Datasets

arXiv: https://arxiv.org/abs/1708.08905 · HTML: https://mlsanity.com/paper/1708.08905/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2017.

## Built on by (most cited first, 5 of 5)

- [Auto-Tables: Synthesizing Multi-Step Transformations to Relationalize Tables without Using Examples](https://mlsanity.com/paper/2307.14565/) (2023)
- [Visual Template Inference for Data Extraction from Documents](https://mlsanity.com/paper/2501.06659/) (2025)
- [AnyMatch -- Efficient Zero-Shot Entity Matching with a Small Language Model](https://mlsanity.com/paper/2409.04073/) (2024)
- [MLOps Spanning Whole Machine Learning Life Cycle: A Survey](https://mlsanity.com/paper/2304.07296/) (2023)
- [Accelerated Cloud for Artificial Intelligence (ACAI)](https://mlsanity.com/paper/2401.16791/) (2024)

## Abstract (arXiv)

Organizations routinely accumulate semi-structured log datasets generated as the output of code; these datasets remain unused and uninterpreted, and occupy wasted space - this phenomenon has been colloquially referred to as "data lake" problem. One approach to leverage these semi-structured datasets is to convert them into a structured relational format, following which they can be analyzed in conjunction with other datasets. We present Datamaran, an tool that extracts structure from semi-structured log datasets with no human supervision. Datamaran automatically identifies field and record endpoints, separates the structured parts from the unstructured noise or formatting, and can tease apart multiple structures from within a dataset, in order to efficiently extract structured relational datasets from semi-structured log datasets, at scale with high accuracy. Compared to other unsupervised log dataset extraction tools developed in prior work, Datamaran does not require the record boundaries to be known beforehand, making it much more applicable to the noisy log files that are ubiquitous in data lakes. Datamaran can successfully extract structured information from all datasets used in prior work, and can achieve 95% extraction accuracy on automatically collected log datasets from GitHub - a substantial 66% increase of accuracy compared to unsupervised schemes from prior work. Our user study further demonstrates that the extraction results of Datamaran are closer to the desired structure than competing algorithms.