mlsanityData to 7 Oct 2026

Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs

arXiv 2409.16341 ↗First posted 24 Sep 2024cs.LG cs.CL cs.SEcited by 17

On the map: LLM agents and tool use → Tool learning and function calling

also in 1 more
Abstract (arXiv)

Training large language models (LLMs) for external tool usage is a rapidly expanding field, with recent research focusing on generating synthetic data to address the shortage of available data. However, the absence of systematic data quality checks poses complications for properly training and testing models. To that end, we propose two approaches for assessing the reliability of data for training LLMs to use external tools. The first approach uses intuitive, human-defined correctness criteria. The second approach uses a model-driven assessment with in-context evaluation. We conduct a thorough evaluation of data quality on two popular benchmarks, followed by an extrinsic evaluation that showcases the impact of data quality on model performance. Our results demonstrate that models trained on high-quality data outperform those trained on unvalidated data, even when trained with a smaller quantity of data. These findings empirically support the significance of assessing and ensuring the reliability of training data for tool-using LLMs.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 13 of its 23 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

13 of 23 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  2. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  3. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  4. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  5. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr

Other references · 8

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023LIMA: Less Is More for Alignmentcited by 1,080
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2020Evaluation of Text Generation: A Surveycited by 275
  5. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  6. 2022RealTime QA: What's the Answer Right Now?cited by 239
  7. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  8. 2018Diversity in Machine Learningcited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025MedArabiQ: Benchmarking Large Language Models on Arabic Medical Taskscited by 15
  2. 2025HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devicescited by 15
  3. 2025GenTool: Enhancing Tool Generalization in Language Models through Zero-to-One and Weak-to-Strong Simulationcited by 9
  4. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 5
  5. 2025Better Datasets Start From RefineLab: Automatic Optimization for High-Quality Dataset Refinementcited by 3
  6. 2026Mi:dm 2.0 Korea-centric Bilingual Language Modelscited by 3
  7. 2026MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmarkcited by 3
  8. 2025Synthetic Data Generation for Phrase Break Prediction with Large Language Modelcited by 2
  9. 2026From Domains to Instances: Dual-Granularity Data Synthesis for LLM Unlearningcited by 2
  10. 2025Rating Quality of Diverse Time Series Data by Meta-learning from LLM Judgmentcited by 1
  11. 2025PuckTrick: A Library for Making Synthetic Data More Realisticcited by 1
  12. 2025ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Executioncited by 1
5 more
  1. 2026SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluationscited by 1
  2. 2026Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agentscited by 1
  3. 2025Enhancing LLMs for Identifying and Prioritizing Important Medical Jargons from Electronic Health Record Notes Utilizing Data Augmentation: A Comparative Studycited by 0
  4. 2026Linguistic and Argument Diversity in Synthetic Data for Function-Calling Agentscited by 0
  5. 2026Adam's Law: Textual Frequency Law on Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
  2. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  3. 2024Hybrid Training Approaches for LLMs: Leveraging Real and Synthetic Data to Enhance Model Performance in Domain-Specific Applications
  4. 2024Efficacy of Synthetic Data as a Benchmark
  5. 2024Evaluating Language Models as Synthetic Data Generators
  6. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  7. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  8. 2024An Empirical Study of Validating Synthetic Data for Formula Generation