# Automatic End-to-End Data Integration using Large Language Models

arXiv: https://arxiv.org/abs/2603.10547 · HTML: https://mlsanity.com/paper/2603.10547/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 11 Mar 2026.

## Builds on (most cited first, 14 of 14)

- [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024)
- [Can Foundation Models Wrangle Your Data?](https://mlsanity.com/paper/2205.09911/) (2022)
- [Match, Compare, or Select? An Investigation of Large Language Models for Entity Matching](https://mlsanity.com/paper/2405.16884/) (2024)
- [Entity Matching using Large Language Models](https://mlsanity.com/paper/2310.11244/) (2023)
- [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025)
- [Using LLMs for the Extraction and Normalization of Product Attribute Values](https://mlsanity.com/paper/2403.02130/) (2024)
- [Magneto: Combining Small and Large Language Models for Schema Matching](https://mlsanity.com/paper/2412.08194/) (2024)
- [ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines](https://mlsanity.com/paper/2504.04808/) (2025)
- [AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Framework](https://mlsanity.com/paper/2412.10422/) (2024)
- [Auto-Pipeline: Synthesizing Complex Data Pipelines By-Target Using Reinforcement Learning and Search](https://mlsanity.com/paper/2106.13861/) (2021)
- [Interactive Data Harmonization with LLM Agents: Opportunities and Challenges](https://mlsanity.com/paper/2502.07132/) (2025)
- [KGpipe: Generation and Evaluation of Pipelines for Data Integration into Knowledge Graphs](https://mlsanity.com/paper/2511.18364/) (2025)
- [FlowETL: An Autonomous Example-Driven Pipeline for Data Engineering](https://mlsanity.com/paper/2507.23118/) (2025)
- [Bridging Business Intent and Data: A Benchmark for Automatic Relational Data Product Generation](https://mlsanity.com/paper/2512.15798/) (2025)

## Built on by (most cited first, 3 of 3)

- [MaDI-Bench: An End-to-End Data Integration Benchmark](https://mlsanity.com/paper/2606.30371/) (2026)
- [Single and Multi Truth Data Fusion using Large Language Models](https://mlsanity.com/paper/2606.28062/) (2026)
- [ArticleMiner: Ontology-Guided Knowledge Graph Construction from Scientific Publications](https://mlsanity.com/paper/2609.25607/) (2026)

## Abstract (arXiv)

Designing data integration pipelines typically requires substantial manual effort from data engineers to configure pipeline components and label training data. While LLMs have shown promise in handling individual steps of the integration process, their potential to replace all human input across end-to-end data integration pipelines has not been investigated. As a step toward exploring this potential, we present an automatic data integration pipeline that uses GPT-5.2 to generate all artifacts required to adapt the pipeline to specific use cases. These artifacts are schema mappings, value mappings for data normalization, training data for entity matching, and validation data for selecting conflict resolution heuristics in data fusion. We compare the performance of this LLM-based pipeline to the performance of human-designed pipelines along three case studies requiring the integration of video game, music, and company related data. Our experiments show that the LLM-based pipeline is able to produce similar results, for some tasks even better results, as the human-designed pipelines. End-to-end, the human and the LLM pipelines produce integrated datasets of comparable size and density. Having the LLM configure the pipelines costs approximately 10 per case study, which represents only a small fraction of the cost of having human data engineers perform the same tasks.