mlsanityData to 14 Oct 2026

Rethinking Data Synthesis: A Teacher Model Training Recipe with Interpretation

arXiv 2410.20362 ↗First posted 27 Oct 2024cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Recent advances in large language model (LLM) training have highlighted the need for diverse, high-quality instruction data. Recently, many works are exploring synthetic data generation using LLMs. However, they primarily focus on prompt engineering with standard supervised instruction-finetuned models, which contains a fundamental limitation: these models are optimized for general question-answering/problem-solving rather than data generation. We propose a paradigm shift named NOMAD by investigating how to specifically train models for data generation, demonstrating that this task differs significantly from training a classical LM. We identify two key factors: no-prompt-masked training and proper training set size selection. Our method, NOMAD, shows substantial improvements over baselines, achieving >4% gains in TriviaQA and >2% in GSM8K with limited training data. Finally, we offer new insights by interpreting synthetic data through the lenses of "relevance" and "novelty".

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 23 references that are in the map. First posted on arXiv 27 Oct 2024.

Builds on

12 of 23 references.

References · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  8. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  9. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  10. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  11. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24
  12. 2024CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learningcited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs
  2. 2024Evaluating Language Models as Synthetic Data Generators
  3. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  4. 2024Prompting-based Synthetic Data Generation for Few-Shot Question Answering
  5. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
  6. 2024Seed-Free Synthetic Data Generation Framework for Instruction-Tuning LLMs: A Case Study in Thai
  7. 2025Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning
  8. 2026It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs