mlsanityData to 7 Oct 2026

Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias

arXiv 2306.15895 ↗First posted 28 Jun 2023cs.CL cs.AI cs.LGcited by 230

Not on the map. Why?

Cited by 37 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have been recently leveraged as training data generators for various natural language processing (NLP) tasks. While previous research has explored different approaches to training models using generated data, they generally rely on simple class-conditional prompts, which may limit the diversity of the generated data and inherit systematic biases of LLM. Thus, we investigate training data generation with diversely attributed prompts (e.g., specifying attributes like length and style), which have the potential to yield diverse and attributed generated data. Our investigation focuses on datasets with high cardinality and diverse domains, wherein we demonstrate that attributed prompts outperform simple class-conditional prompts in terms of the resulting model's performance. Additionally, we present a comprehensive empirical study on data generation encompassing vital aspects like bias, diversity, and efficiency, and highlight three key observations: firstly, synthetic datasets generated by simple prompts exhibit significant biases, such as regional bias; secondly, attribute diversity plays a pivotal role in enhancing model performance; lastly, attributed prompts achieve the performance of simple class-conditional prompts while utilizing only 5% of the querying cost of ChatGPT associated with the latter. The data and code are available on urlhttps://github.com/yueyu1030/AttrPrompt.

Key facts. Cited by 230 papers in the map (data through 7 Oct 2026). Builds on 54 of its 68 references that are in the map. First posted on arXiv 28 Jun 2023.

Builds on

54 of 68 references.

References · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
38 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  2. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  3. 2019When Does Label Smoothing Help?cited by 985
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  5. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  6. 2023Instruction Tuning with GPT-4cited by 676
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  8. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  9. 2023Textbooks Are All You Needcited by 561
  10. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  11. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  12. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  13. 2021True Few-Shot Learning with Language Modelscited by 343
  14. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  15. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  16. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  17. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  18. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  19. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  20. 2021Generating Datasets with Pretrained Language Modelscited by 192
  21. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  22. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  23. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  24. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 127
  25. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  26. 2019On the Use of ArXiv as a Datasetcited by 101
  27. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  28. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
  29. 2022RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extractioncited by 60
  30. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  31. 2018Content preserving text generation with attribute controlscited by 47
  32. 2021RAFT: A Real-World Few-Shot Text Classification Benchmarkcited by 44
  33. 2018Weakly-Supervised Hierarchical Text Classificationcited by 42
  34. 2021TWEAC: Transformer with Extendable QA Agent Classifierscited by 37
  35. 2021Attribute Alignment: Controlling Text Generation from Pre-trained Language Modelscited by 27
  36. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  37. 2020DQI: Measuring Data Quality in NLPcited by 23
  38. 2023Mixture of Soft Prompts for Controllable Data Generationcited by 23

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 104; 2025: 81; 2026: 24.

230 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  3. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  4. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  5. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  6. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  7. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 150
  8. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  9. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  10. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  11. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  12. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
38 more
  1. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
  2. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  3. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  4. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57
  5. 2024TinyAgent: Function Calling at the Edgecited by 55
  6. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 54
  7. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  8. 2024Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Datacited by 52
  9. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  10. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  11. 2025Diverse Preference Optimizationcited by 48
  12. 2024DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflowscited by 43
  13. 2023Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matchingcited by 40
  14. 2024CodecLM: Aligning Language Models with Tailored Synthetic Datacited by 39
  15. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  16. 2024AI-Assisted Generation of Difficult Math Questionscited by 37
  17. 2024Improving the Robustness of Large Language Models via Consistency Alignmentcited by 36
  18. 2024AI and the Problem of Knowledge Collapsecited by 34
  19. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  20. 2024StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examplescited by 32
  21. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 32
  22. 2023Aligning Large Language Models with Human: A Surveycited by 31
  23. 2023DialogBench: Evaluating LLMs as Human-like Dialogue Systemscited by 31
  24. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 31
  25. 2024Self-Pluralising Culture Alignment for Large Language Modelscited by 31
  26. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  27. 2023Distilled GPT for Source Code Summarizationcited by 30
  28. 2024Conformity in Large Language Modelscited by 30
  29. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  30. 2024TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervisioncited by 25
  31. 2024Privacy Preserving Prompt Engineering: A Surveycited by 24
  32. 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
  33. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 23
  34. 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
  35. 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attackscited by 23
  36. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 22
  37. 2025Multi-Agent Simulator Drives Language Models for Legal Intensive Interactioncited by 22
  38. 2024A Survey on Natural Language Counterfactual Generationcited by 21

50 most cited of 230 All 230 (JSON)