mlsanityData to 7 Oct 2026

Synthetic Data for any Differentiable Target

arXiv 2604.08423 ↗First posted 9 Apr 2026cs.CL cs.AI cs.LGcited by 2

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

What are the limits of controlling language models via synthetic training data? We develop a reinforcement learning (RL) primitive, the Dataset Policy Gradient (DPG), which can precisely optimize synthetic data generators to produce a dataset of targeted examples. When used for supervised fine-tuning (SFT) of a target model, these examples cause the target model to do well on a differentiable metric of our choice. Our approach achieves this by taking exact data attribution via higher-order gradients and using those scores as policy gradient rewards. We prove that this procedure closely approximates the true, intractable gradient for the synthetic data generator. To illustrate the potential of DPG, we show that, using only SFT on generated examples, we can cause the target model's LM head weights to (1) embed a QR code, (2) embed the pattern texttt67, and (3) have lower ℓ2 norm. We additionally show that we can cause the generator to (4) rephrase inputs in a new language and (5) produce a specific UUID, even though neither of these objectives is conveyed in the generator's input prompts. These findings suggest that DPG is a powerful and flexible technique for shaping model properties using only synthetic training examples.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 32 of its 50 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

32 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Synthetic bootstrapped pretraining17 Sep

Other references · 31

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  7. 2024Phi-4 Technical Reportcited by 675
  8. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  9. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  10. 2018Learning Word Vectors for 157 Languagescited by 279
  11. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  12. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
19 more
  1. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  2. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  3. 2020MetaPoison: Practical General-purpose Clean-label Data Poisoningcited by 114
  4. 2019Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Datacited by 103
  5. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  6. 2025Self-Adapting Language Modelscited by 71
  7. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
  8. 2024Synthetic continued pretrainingcited by 56
  9. 2025Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMscited by 47
  10. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  11. 2025Reasoning to Learn from Latent Thoughtscited by 43
  12. 2021Meta-Learning to Improve Pre-Trainingcited by 27
  13. 2025Optimizing ML Training with Metagradient Descentcited by 22
  14. 2025DataRater: Meta-Learned Dataset Curationcited by 13
  15. 2025Blackbox Model Provenance via Palimpsestic Membership Inferencecited by 11
  16. 2025MixMin: Finding Data Mixtures via Convex Minimizationcited by 10
  17. 2025Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputscited by 7
  18. 2026Infusion: Shaping Model Behavior by Editing Training Data via Influence Functionscited by 4
  19. 2026Test-Time Meta-Adaptation with Self-Synthesiscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Subliminal Steering: Stronger Encoding of Hidden Signalscited by 10
  2. 2026Self-Play Pretraining with Zero Datacited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning
  2. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation
  3. 2025Towards a Unified View of Large Language Model Post-Training
  4. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
  5. 2024SoftSRV: Learn to Generate Targeted Synthetic Data
  6. 2025Synthetic Data RL: Task Definition Is All You Need
  7. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)
  8. 2025Multi-Sample Prompting and Actor-Critic Prompt Optimization for Diverse Synthetic Data Generation