mlsanityData to 7 Oct 2026

Invent a Dataset: Measuring dataset generation abilities with zero seed

arXiv 2610.01674 ↗First posted 1 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Building datasets remains one of the most manual and brittle parts of AI development. In this technical report, we focus on the most extreme but also most prevalent setting real world practitioners face: a zero data regime. Here, practitioners don't have any data for the capability they want to learn. We introduce Invent-A-Dataset which is a prompt based system to go from dataset description to realistic and large scale post-training datasets. We evaluate Invent-A-Dataset against five frontier model APIs including Anthropic, Google, Open AI, DeepSeek, Zai. Across eight task types and dataset sizes up to 20K samples, Invent-A-Dataset significantly outperforms with both the highest quality (17% relative gains) while simultaneously producing the most diverse samples (19% relative gains). Its diversity advantage widens with scale of training dataset size (from parity at 200 samples to 37% relative gains at 20K samples). This translates into considerable downstream training gains, resulting in far more performant post-trained models. Invent-A-Dataset fine-tune consistently ranks higher compared to other generator fine-tunes across different post-trained model architectures.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 66 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

43 of 66 references.

References · 43

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  6. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  7. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2022Galactica: A Large Language Model for Sciencecited by 736
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  12. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
31 more
  1. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  2. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  3. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  4. 2024A Survey on Data Selection for Language Modelscited by 230
  5. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  6. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  7. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 181
  8. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  9. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  10. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  11. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 112
  12. 2018A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspectivecited by 111
  13. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  14. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
  15. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 90
  16. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  17. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  18. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  19. 2020On the Value of Out-of-Distribution Testing: An Example of Goodhart's Lawcited by 74
  20. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  21. 2024CodecLM: Aligning Language Models with Tailored Synthetic Datacited by 39
  22. 2023Self-Influence Guided Data Reweighting for Language Model Pre-trainingcited by 37
  23. 2019Dataset search: a surveycited by 36
  24. 2023Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluationcited by 18
  25. 2026Autodata: An agentic data scientist to create high quality synthetic datacited by 12
  26. 2024Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruningcited by 10
  27. 2025DatasetResearch: Benchmarking Agent Systems for Demand-Driven Dataset Discoverycited by 9
  28. 2025Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluationcited by 6
  29. 2025A Survey on Open Dataset Search in the LLM Era: Retrospectives and Perspectivescited by 2
  30. 2025Rethinking Dataset Discovery with DataScoutcited by 1
  31. 2025Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuningcited by 1

Built on by

Not cited yet.