mlsanityData to 7 Oct 2026

Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning

arXiv 2605.30039 ↗First posted 28 May 2026cs.AIcited by 0

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Large Language Models have demonstrated remarkable progress in general-purpose capabilities and can achieve strong performance in specific domains through fine-tuning on domain-specific data. However, acquiring high-quality data for target domains remains a significant challenge. Existing data synthesis approaches follow a deductive paradigm, heavily relying on explicit domain descriptions expressed in natural language and careful prompt engineering, limiting their applicability in real-world scenarios where domains are difficult to describe or formally articulate. In this work, we tackle the underexplored problem of domain-specific data synthesis through an inductive paradigm, where the target domain is defined only through a set of reference examples, particularly when domain characteristics are difficult to articulate in natural language. We propose a novel framework, DOMINO, that learns a minimal sufficient domain representation from reference samples and leverages it to guide the generation of domain-aligned synthetic data. DOMINO integrates prompt tuning with a contrastive disentanglement objective to separate domain-level patterns from sample-specific noise, mitigating overfitting while preserving core domain characteristics. Theoretically, we prove that DOMINO expands the support of the synthetic data distribution, ensuring greater diversity. Empirically, on challenging coding benchmarks where domain definitions are implicit, fine-tuning on data synthesized by DOMINO improves Pass@1 accuracy by up to 4.63% over strong, instruction-tuned backbones, demonstrating its effectiveness and robustness. This work establishes a new paradigm for domain-specific data synthesis, enabling practical and scalable domain adaptation without manual prompt design or natural language domain specifications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 23 of its 30 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

23 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Models27 Jun
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey14 Jun

Other references · 21

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  12. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
9 more
  1. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  2. 2025LIMO: Less is More for Reasoningcited by 488
  3. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  4. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  5. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  6. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  7. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoningcited by 58
  8. 2024MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Datacited by 43
  9. 2025LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selectioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DS2-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning
  2. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation
  3. 2025Building Domain-Specific Small Language Models via Guided Data Generation
  4. 2024A synthetic data approach for domain generalization of NLI models
  5. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  6. 2026It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
  7. 2024DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models
  8. 2024A Survey on Data Synthesis and Augmentation for Large Language Models