# DataEvolver: Automatic Data Preparation for Large Language Models through Multi-Level Self-Evolving

arXiv: https://arxiv.org/abs/2606.07001 · HTML: https://mlsanity.com/paper/2606.07001/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Code and instruction data synthesis](https://mlsanity.com/synth/code-and-instruction-data-synthesis/)

## Builds on (most cited first, 29 of 29)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining](https://mlsanity.com/paper/2305.10429/) (2023)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://mlsanity.com/paper/2303.09540/) (2023)
- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems](https://mlsanity.com/paper/2508.07407/) (2025)
- [MoDS: Model-oriented Data Selection for Instruction Tuning](https://mlsanity.com/paper/2311.15653/) (2023)
- [Symbolic Learning Enables Self-Evolving Agents](https://mlsanity.com/paper/2406.18532/) (2024)
- [AlphaClean: Automatic Generation of Data Cleaning Pipelines](https://mlsanity.com/paper/1904.11827/) (2019)
- [Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models](https://mlsanity.com/paper/2501.14755/) (2024)
- [Language Models as Continuous Self-Evolving Data Engineers](https://mlsanity.com/paper/2412.15151/) (2024)
- [Text-to-Pipeline: Bridging Natural Language and Data Preparation Pipelines](https://mlsanity.com/paper/2505.15874/) (2025)
- [DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining](https://mlsanity.com/paper/2410.00260/) (2024)
- [Prompt2DAG: A Modular Methodology for LLM-Based Data Enrichment Pipeline Generation](https://mlsanity.com/paper/2509.13487/) (2025)
- [LLaPipe: LLM-Guided Reinforcement Learning for Automated Data Preparation Pipeline Construction](https://mlsanity.com/paper/2507.13712/) (2025)

## Built on by (most cited first, 3 of 3)

- [Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops](https://mlsanity.com/paper/2607.07663/) (2026)
- [SkillAdam: Stable and Efficient Skill Evolution for Agents](https://mlsanity.com/paper/2609.08944/) (2026)
- [What is Missing from AI Post-Training AI: An Empirical Analysis](https://mlsanity.com/paper/2608.19072/) (2026)

## Abstract (arXiv)

High-quality training data is essential to large language models (LLMs) and typically requires extensive and costly manual curation. Automatic data preparation systems support reusable operators and task-driven workflows, yet adapting these workflows to the desired data quality remains challenging. High-quality examples provide a practical specification for guiding this adaptation. We introduce DataEvolver, a data preparation system that automatically constructs pipelines through multi-level self-evolution. At the operator level, it expands the operator library and refines logical plans to resolve dependency conflicts. At the pipeline level, it instantiates these plans into code and uses trial-output feedback to refine the pipeline toward seed-specified quality requirements. Experiments on seven benchmarks show that DataEvolver substantially improves data quality and achieves an average gain of approximately 10 score points in downstream LLM performance compared with training on original data, highlighting new opportunities for the iterative co-evolution of LLMs and data.