# Automated Data Curation for Robust Language Model Fine-Tuning

arXiv: https://arxiv.org/abs/2403.12776 · HTML: https://mlsanity.com/paper/2403.12776/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Ask Me Anything: Dynamic Memory Networks for Natural Language Processing](https://mlsanity.com/paper/1506.07285/) (2015)
- [Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences](https://mlsanity.com/paper/2401.10529/) (2024)
- [DataPerf: Benchmarks for Data-Centric AI Development](https://mlsanity.com/paper/2207.10062/) (2022)
- [The Turking Test: Can Language Models Understand Instructions?](https://mlsanity.com/paper/2010.11982/) (2020)
- [How Many Demonstrations Do You Need for In-context Learning?](https://mlsanity.com/paper/2303.08119/) (2023)
- [Gaussian Process Assisted Active Learning of Physical Laws](https://mlsanity.com/paper/1910.03120/) (2019)
- [Why Propagate Alone? Parallel Use of Labels and Features on Graphs](https://mlsanity.com/paper/2110.07190/) (2021)
- [Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features](https://mlsanity.com/paper/2110.13413/) (2021)
- [A Robust Stacking Framework for Training Deep Graph Models with Multifaceted Node Features](https://mlsanity.com/paper/2206.08473/) (2022)

## Built on by (most cited first, 27 of 27)

- [Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement](https://mlsanity.com/paper/2405.15973/) (2024)
- [FineVision: Open Data Is All You Need](https://mlsanity.com/paper/2510.17269/) (2025)
- [Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning](https://mlsanity.com/paper/2502.01968/) (2025)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [Improving Data Efficiency via Curating LLM-Driven Rating Systems](https://mlsanity.com/paper/2410.10877/) (2024)
- [Residual-based Language Models are Free Boosters for Biomedical Imaging](https://mlsanity.com/paper/2403.17343/) (2024)
- [Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling](https://mlsanity.com/paper/2410.03735/) (2024)
- [Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs](https://mlsanity.com/paper/2505.16967/) (2025)
- [AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement](https://mlsanity.com/paper/2608.20318/) (2026)
- [ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning](https://mlsanity.com/paper/2412.00631/) (2024)
- [A Survey on Efficient Large Language Model Training: From Data-centric Perspectives](https://mlsanity.com/paper/2510.25817/) (2025)
- [TokenUnify: Scaling Up Autoregressive Pretraining for Neuron Segmentation](https://mlsanity.com/paper/2405.16847/) (2024)
- [Multi-Objective Linguistic Control of Large Language Models](https://mlsanity.com/paper/2406.16229/) (2024)
- [Automatic register identification for the open web using multilingual deep learning](https://mlsanity.com/paper/2406.19892/) (2024)
- [Video-to-Text Pedestrian Monitoring (VTPM): Leveraging Computer Vision and Large Language Models for Privacy-Preserve Pedestrian Activity Monitoring at Intersections](https://mlsanity.com/paper/2408.11649/) (2024)
- [CSRec: Rethinking Sequential Recommendation from A Causal Perspective](https://mlsanity.com/paper/2409.05872/) (2024)
- [Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights](https://mlsanity.com/paper/2501.06708/) (2025)
- [Practical Principles for AI Cost and Compute Accounting](https://mlsanity.com/paper/2502.15873/) (2025)
- [Data-Centric AI in the Age of Large Language Models](https://mlsanity.com/paper/2406.14473/) (2024)
- [Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in Large Language Models](https://mlsanity.com/paper/2410.12989/) (2024)
- [AutoPureData: Automated Filtering of Undesirable Web Data to Update LLM Knowledge](https://mlsanity.com/paper/2406.19271/) (2024)
- [Self-Rationalization in the Wild: A Large Scale Out-of-Distribution Evaluation on NLI-related tasks](https://mlsanity.com/paper/2502.04797/) (2025)
- [An Empirical Study of Validating Synthetic Data for Formula Generation](https://mlsanity.com/paper/2407.10657/) (2024)
- [Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments](https://mlsanity.com/paper/2412.12417/) (2024)
- [Real-time prediction of breast cancer sites using deformation-aware graph neural network](https://mlsanity.com/paper/2511.13082/) (2025)
- [Uncovering Competency Gaps in Large Language Models and Their Benchmarks](https://mlsanity.com/paper/2512.20638/) (2025)
- [Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxies](https://mlsanity.com/paper/2605.09665/) (2026)

## Abstract (arXiv)

Large Language Models have become the de facto approach to sequence-to-sequence text generation tasks, but for specialized tasks/domains, a pretrained LLM lacks specific capabilities to produce accurate or well-formatted responses. Supervised fine-tuning specializes a LLM by training it on dataset of example prompts with target responses, but real-world data tends to be noisy. While many fine-tuning algorithms exist, here we consider a data-centric AI perspective on LLM fine-tuning, studying how to systematically curate the training dataset to improve the LLM produced via any fine-tuning algorithm. We introduce an automated data curation pipeline CLEAR (Confidence-based LLM Evaluation And Rectification) for instruction tuning datasets, that can be used with any LLM and fine-tuning procedure. CLEAR estimates which training data is low-quality and either filters or corrects it. Automatically identifying which data to filter or correct is done via LLM-derived confidence estimates, to ensure only confident modifications to the dataset. Unlike existing data curation techniques, CLEAR is a comprehensive framework that can improve a dataset (and trained model outputs) without additional fine-tuning computations. We don't assume access to a stronger LLM than the model being fine-tuned (e.g. relying on GPT-4 when fine-tuning GPT-3.5), to see whether CLEAR can meaningfully improve the capabilities of any LLM. Experiments reveal that CLEAR consistently improves the performance of fine-tuned models across many datasets and models (like GPT-3.5 and Llama2).