# Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs

arXiv: https://arxiv.org/abs/2409.16341 · HTML: https://mlsanity.com/paper/2409.16341/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)
- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 13 of 13)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions](https://mlsanity.com/paper/2306.02224/) (2023)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [Diversity in Machine Learning](https://mlsanity.com/paper/1807.01477/) (2018)

## Built on by (most cited first, 19 of 19)

- [MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks](https://mlsanity.com/paper/2505.03427/) (2025)
- [HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices](https://mlsanity.com/paper/2505.19628/) (2025)
- [GenTool: Enhancing Tool Generalization in Language Models through Zero-to-One and Weak-to-Strong Simulation](https://mlsanity.com/paper/2502.18990/) (2025)
- [Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence](https://mlsanity.com/paper/2510.16657/) (2025)
- [Mi:dm 2.0 Korea-centric Bilingual Language Models](https://mlsanity.com/paper/2601.09066/) (2026)
- [Better Datasets Start From RefineLab: Automatic Optimization for High-Quality Dataset Refinement](https://mlsanity.com/paper/2511.06530/) (2025)
- [MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark](https://mlsanity.com/paper/2602.01714/) (2026)
- [Synthetic Data Generation for Phrase Break Prediction with Large Language Model](https://mlsanity.com/paper/2507.18044/) (2025)
- [From Domains to Instances: Dual-Granularity Data Synthesis for LLM Unlearning](https://mlsanity.com/paper/2601.04278/) (2026)
- [SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations](https://mlsanity.com/paper/2605.22564/) (2026)
- [Rating Quality of Diverse Time Series Data by Meta-learning from LLM Judgment](https://mlsanity.com/paper/2506.01290/) (2025)
- [PuckTrick: A Library for Making Synthetic Data More Realistic](https://mlsanity.com/paper/2506.18499/) (2025)
- [ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution](https://mlsanity.com/paper/2507.15501/) (2025)
- [Linguistic and Argument Diversity in Synthetic Data for Function-Calling Agents](https://mlsanity.com/paper/2601.17829/) (2026)
- [Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents](https://mlsanity.com/paper/2608.06329/) (2026)
- [Enhancing LLMs for Identifying and Prioritizing Important Medical Jargons from Electronic Health Record Notes Utilizing Data Augmentation: A Comparative Study](https://mlsanity.com/paper/2502.16022/) (2025)
- [Adam's Law: Textual Frequency Law on Large Language Models](https://mlsanity.com/paper/2604.02176/) (2026)
- [Metric-Construction Coupling Inflates Measured Synthetic Dialect Recovery](https://mlsanity.com/paper/2610.00164/) (2026)
- [Turnslide: Scalable Multi-Turn Data Synthesis by Walking a Finite-State Machine](https://mlsanity.com/paper/2610.07070/) (2026)

## Abstract (arXiv)

Training large language models (LLMs) for external tool usage is a rapidly expanding field, with recent research focusing on generating synthetic data to address the shortage of available data. However, the absence of systematic data quality checks poses complications for properly training and testing models. To that end, we propose two approaches for assessing the reliability of data for training LLMs to use external tools. The first approach uses intuitive, human-defined correctness criteria. The second approach uses a model-driven assessment with in-context evaluation. We conduct a thorough evaluation of data quality on two popular benchmarks, followed by an extrinsic evaluation that showcases the impact of data quality on model performance. Our results demonstrate that models trained on high-quality data outperform those trained on unvalidated data, even when trained with a smaller quantity of data. These findings empirically support the significance of assessing and ensuring the reliability of training data for tool-using LLMs.