# Making Large Language Models Better Data Creators

arXiv: https://arxiv.org/abs/2310.20111 · HTML: https://mlsanity.com/paper/2310.20111/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 31 Oct 2023.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge](https://mlsanity.com/paper/2109.01653/) (2021)
- [Towards Zero-Label Language Learning](https://mlsanity.com/paper/2109.09193/) (2021)
- [RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge](https://mlsanity.com/paper/2101.00376/) (2021)

## Built on by (most cited first, 28 of 28)

- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance](https://mlsanity.com/paper/2401.03729/) (2024)
- [Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation](https://mlsanity.com/paper/2402.11443/) (2024)
- [Faithful Persona-based Conversational Dataset Generation with Large Language Models](https://mlsanity.com/paper/2312.10007/) (2023)
- [A Comprehensive Evaluation of Cognitive Biases in LLMs](https://mlsanity.com/paper/2410.15413/) (2024)
- [TnT-LLM: Text Mining at Scale with Large Language Models](https://mlsanity.com/paper/2403.12173/) (2024)
- [AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs](https://mlsanity.com/paper/2502.01977/) (2025)
- ["Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?](https://mlsanity.com/paper/2406.17600/) (2024)
- [Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks](https://mlsanity.com/paper/2402.13482/) (2024)
- [ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support](https://mlsanity.com/paper/2602.01885/) (2026)
- [Are Chatbots Reliable Text Annotators? Sometimes](https://mlsanity.com/paper/2311.05769/) (2023)
- [Towards Enhanced Immersion and Agency for LLM-based Interactive Drama](https://mlsanity.com/paper/2502.17878/) (2025)
- [Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following](https://mlsanity.com/paper/2512.23457/) (2025)
- [1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data](https://mlsanity.com/paper/2408.03506/) (2024)
- [Automatic Labelling with Open-source LLMs using Dynamic Label Schema Integration](https://mlsanity.com/paper/2501.12332/) (2025)
- [LLM in the Loop: Creating the ParaDeHate Dataset for Hate Speech Detoxification](https://mlsanity.com/paper/2506.01484/) (2025)
- [Approximating Language Model Training Data from Weights](https://mlsanity.com/paper/2506.15553/) (2025)
- [Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs' Instruction Following Capability](https://mlsanity.com/paper/2504.21625/) (2025)
- [Assessing the quality of information extraction](https://mlsanity.com/paper/2404.04068/) (2024)
- [PublicAgent: Multi-Agent Design Principles From an LLM-Based Open Data Analysis Framework](https://mlsanity.com/paper/2511.03023/) (2025)
- [Fine-grainedly Synthesize Streaming Data Based On Large Language Models With Graph Structure Understanding For Data Sparsity](https://mlsanity.com/paper/2403.06139/) (2024)
- [USDC: A Dataset of underlineUser underlineStance and underlineDogmatism in Long underlineConversations](https://mlsanity.com/paper/2406.16833/) (2024)
- [Forecasting Future International Events: A Reliable Dataset for Text-Based Event Modeling](https://mlsanity.com/paper/2411.14042/) (2024)
- [Multimodal Behavioral Patterns Analysis with Eye-Tracking and LLM-Based Reasoning](https://mlsanity.com/paper/2507.18252/) (2025)
- [On the Necessity of World Knowledge for Mitigating Missing Labels in Extreme Classification](https://mlsanity.com/paper/2408.09585/) (2024)
- [Chinese Essay Rhetoric Recognition Using LoRA, In-context Learning and Model Ensemble](https://mlsanity.com/paper/2604.14167/) (2026)
- [An Agentic Hybrid Top-Down and Bottom-Up Approach to Knowledge Graph Generation](https://mlsanity.com/paper/2608.07023/) (2026)
- [TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding](https://mlsanity.com/paper/2609.01810/) (2026)

## Abstract (arXiv)

Although large language models (LLMs) have advanced the state-of-the-art in NLP significantly, deploying them for downstream applications is still challenging due to cost, responsiveness, control, or concerns around privacy and security. As such, trainable models are still the preferred option in some cases. However, these models still require human-labeled data for optimal performance, which is expensive and time-consuming to obtain. In order to address this issue, several techniques to reduce human effort involve labeling or generating data using LLMs. Although these methods are effective for certain applications, in practice they encounter difficulties in real-world scenarios. Labeling data requires careful data selection, while generating data necessitates task-specific prompt engineering. In this paper, we propose a unified data creation pipeline that requires only a single formatting example, and which is applicable to a broad range of tasks, including traditionally problematic ones with semantically devoid label spaces. In our experiments we demonstrate that instruction-following LLMs are highly cost-effective data creators, and that models trained with these data exhibit performance better than those trained with human-labeled data (by up to 17.5%) on out-of-distribution evaluation, while maintaining comparable performance on in-distribution tasks. These results have important implications for the robustness of NLP systems deployed in the real-world.