mlsanityData to 14 Oct 2026

SynQuE: Estimating Synthetic Dataset Quality Without Annotations

arXiv 2511.03928 ↗First posted 6 Nov 2025cs.LGcited by 2

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

We introduce and formalize the Synthetic Dataset Quality Estimation (SynQuE) problem: ranking synthetic datasets by their expected real-world task performance using only limited unannotated real data. This addresses a critical and open challenge where data is scarce due to collection costs or privacy constraints. We establish the first comprehensive benchmarks for this problem by introducing and evaluating proxy metrics that choose synthetic data for training to maximize task performance on real data. We introduce the first proxy metrics for SynQuE by adapting distribution and diversity-based distance measures to our context via embedding models. To address the shortcomings of these metrics on complex planning tasks, we propose LENS, a novel proxy that leverages large language model reasoning. Our results show that SynQuE proxies correlate with real task performance across diverse tasks, including sentiment analysis, Text2SQL, web navigation, and image classification, with LENS consistently outperforming others on complex tasks by capturing nuanced characteristics. For instance, on text-to-SQL parsing, training on the top-3 synthetic datasets selected via SynQuE proxies can raise accuracy from 30.4% to 38.4 (+8.1)% on average compared to selecting data indiscriminately. This work establishes SynQuE as a practical framework for synthetic data selection under real-data scarcity and motivates future research on foundation model-based data characterization and fine-grained data selection. We release our code.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 58 of its 81 references that are in the map. First posted on arXiv 6 Nov 2025.

Builds on

58 of 81 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code18 Mar
  2. 2025Measuring Diversity in Synthetic Datasets12 Feb
  3. 2024Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification28 Oct
  4. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey14 Jun
  5. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations11 Oct

Other references · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  9. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  10. 2024A Survey on LLM-as-a-Judgecited by 1,423
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  12. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
38 more
  1. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  2. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  3. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  4. 2019BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learningcited by 432
  5. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  6. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  7. 2015Generative Moment Matching Networkscited by 257
  8. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 256
  9. 2023Self-Consuming Generative Models Go MADcited by 250
  10. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  11. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
  12. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  13. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  14. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  15. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  16. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
  17. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
  18. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  19. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
  20. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
  21. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 126
  22. 2022BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotationscited by 99
  23. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  24. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 83
  25. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 79
  26. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  27. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 59
  28. 2024Synthesizing Text-to-SQL Data from Weak and Strong LLMscited by 54
  29. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
  30. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 50
  31. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  32. 2018Faster k-Medoids Clustering: Improving the PAM, CLARA, and CLARANS Algorithmscited by 44
  33. 2025DataDecide: How to Predict Best Pretraining Data with Small Experimentscited by 40
  34. 2019Dataset search: a surveycited by 36
  35. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  36. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  37. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 32
  38. 2020Combating Mode Collapse in GAN training: An Empirical Analysis using Hessian Eigenvaluescited by 27

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontiercited by 3
  2. 2026CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Setscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models
  2. 2024Efficacy of Synthetic Data as a Benchmark
  3. 2025Scaling Laws of Synthetic Data for Language Models
  4. 2026Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility
  5. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  6. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Models
  7. 2024A Survey of Data Synthesis Approaches
  8. 2024CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation