mlsanityData to 7 Oct 2026

Training-Aware Target Coverage for Synthetic Data Selection

arXiv 2610.00814 ↗First posted 30 Sep 2026cs.LG cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Synthetic data are increasingly used to scale LLM training, yet more synthetic data do not necessarily produce better models. Useful synthetic data must add information relevant to the target task without introducing errors that offset their benefit, and the value of an example can change as the training set grows. We develop a linear theory that characterizes this tradeoff and determines where synthetic data are useful, how much should be added, and the marginal value of adding one example to an existing set. The analysis shows the conditions when input coverage alone is sufficient and when synthetic errors must also be considered. Guided by these results, we introduce Training-Aware Target Coverage (TATC), a synthetic data selection method for LLM fine-tuning. TATC identifies candidates whose training effects are beneficial to the target task and selects among them to expand coverage of target-relevant directions not already represented by the available data. Experiments on text and image data verify the linear theory. With mathematical reasoning tasks, TATC selects synthetic solutions for fine-tuning Qwen2.5-Math-1.5B-Instruct and outperforms alternative synthetic-data selection methods on GSM8K across selection budgets. In summary, we provide a principled approach to synthetic data selection by quantifying and maximizing its value to the target task.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 29 of its 36 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

29 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Optimsyn: Influence-Guided Rubrics Optimization for Synthetic Data Generation1 Apr
  2. 2024Fill In The Gaps: Model Calibration and Generalization with Synthetic Data7 Oct
  3. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personas28 Jun

Other references · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  9. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  10. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  11. 2023Textbooks Are All You Needcited by 561
  12. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
14 more
  1. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  2. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  3. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  4. 2023Self-Consuming Generative Models Go MADcited by 239
  5. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  6. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  7. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  8. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  9. 2021Gone Fishing: Neural Active Learning with Fisher Embeddingscited by 88
  10. 2025Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoningcited by 38
  11. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  12. 2025High-dimensional Analysis of Synthetic Data Selectioncited by 5
  13. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 5
  14. 2026SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Trainingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  2. 2026Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility
  3. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation
  4. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  5. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  6. 2025Less is More: Adaptive Coverage for Synthetic Training Data
  7. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
  8. 2025RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Sets