mlsanityData to 14 Oct 2026

Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Science

arXiv 2305.15041 ↗First posted 24 May 2023cs.CLcited by 37

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Large Language Models (LLMs) have democratized synthetic data generation, which in turn has the potential to simplify and broaden a wide gamut of NLP tasks. Here, we tackle a pervasive problem in synthetic data generation: its generative distribution often differs from the distribution of real-world data researchers care about (in other words, it is unfaithful). In a case study on sarcasm detection, we study three strategies to increase the faithfulness of synthetic data: grounding, filtering, and taxonomy-based generation. We evaluate these strategies using the performance of classifiers trained with generated synthetic data on real-world data. While all three strategies improve the performance of classifiers, we find that grounding works best for the task at hand. As synthetic data generation plays an ever-increasing role in NLP research, we expect this work to be a stepping stone in improving its utility. We conclude this paper with some recommendations on how to generate high(er)-fidelity synthetic data for specific tasks.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 20 of its 26 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

20 of 26 references.

References · 20

  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  5. 2023LIMA: Less Is More for Alignmentcited by 1,100
  6. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  7. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  8. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  9. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  10. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  11. 2021A Survey of Data Augmentation Approaches for NLPcited by 409
  12. 2023Can Large Language Models Transform Computational Social Science?cited by 250
8 more
  1. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 196
  2. 2019Not Enough Data? Deep Learning to the Rescue!cited by 175
  3. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 129
  4. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  5. 2020DARE: Data Augmented Relation Extraction with GPT-2cited by 43
  6. 2019iSarcasm: A Dataset of Intended Sarcasmcited by 41
  7. 2023The Parrot Dilemma: Human-Labeled vs. LLM-augmented Data in Classification Taskscited by 35
  8. 2022UTNLP at SemEval-2022 Task 6: A Comparative Analysis of Sarcasm Detection Using Generative-based and Mutation-based Data Augmentationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 15; 2025: 12; 2026: 3.

37 papers.

Sort:

By area:

  1. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 103
  2. 2025LLM Social Simulations Are a Promising Research Methodcited by 99
  3. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 57
  4. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 47
  5. 2024LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptionscited by 47
  6. 2023People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detectioncited by 25
  7. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  8. 2024The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussionscited by 16
  9. 2025Language Agents as Digital Representatives in Collective Decision-Makingcited by 16
  10. 2024LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documentscited by 12
  11. 2024Data Generation Using Large Language Models for Text Classification: An Empirical Case Studycited by 11
  12. 2024DiffLM: Controllable Synthetic Data Generation via Diffusion Language Modelscited by 9
25 more
  1. 2024When Reasoning Meets Information Aggregation: A Case Study with Sports Narrativescited by 7
  2. 2025The Accountability Paradox: How Platform API Restrictions Undermine AI Transparency Mandatescited by 7
  3. 2025Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentencescited by 5
  4. 2024Curating Grounded Synthetic Data with Global Perspectives for Equitable AIcited by 4
  5. 2024Aligning Actions and Walking to LLM-Generated Textual Descriptionscited by 3
  6. 2024Towards Realistic Synthetic User-Generated Content: A Scaffolding Approach to Generating Online Discussionscited by 3
  7. 2025Filtering with Confidence: When Data Augmentation Meets Conformal Predictioncited by 3
  8. 2023Steering Language Generation: Harnessing Contrastive Expert Guidance and Negative Prompting for Coherent and Diverse Synthetic Data Generationcited by 2
  9. 2024Improving Clinical NLP Performance through Language Model-Generated Synthetic Clinical Datacited by 2
  10. 2024MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representationscited by 2
  11. 2024In-Context Learning for Preserving Patient Privacy: A Framework for Synthesizing Realistic Patient Portal Messagescited by 2
  12. 2025Can Large Language Models Generate Effective Datasets for Emotion Recognition in Conversations?cited by 2
  13. 2025HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generationcited by 2
  14. 2023Relation Extraction in underexplored biomedical domains: A diversity-optimised sampling and synthetic data generation approachcited by 1
  15. 2024Agentic AI: The Era of Semantic Decodingcited by 1
  16. 2025Reliability of Large Language Model Generated Clinical Reasoning in Assisted Reproductive Technology: Blinded Comparative Evaluation Studycited by 1
  17. 2023Releasing the CRaQAn (Coreference Resolution in Question-Answering): An open-source dataset and dataset creation methodology using instruction-following modelscited by 0
  18. 2024Adaptable Embeddings Network (AEN)cited by 0
  19. 2025Survey Transfer Learning: Recycling Data with Silicon Responsescited by 0
  20. 2025Towards High-Fidelity Synthetic Multi-platform Social Media Datasets via Large Language Modelscited by 0
  21. 2025The Synthetic Mirror -- Synthetic Data at the Age of Agentic AIcited by 0
  22. 2025ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Linkscited by 0
  23. 2026Agents of Diffusion: Enhancing Diffusion Language Models with Multi-Agent Reinforcement Learning for Structured Data Generation (Extended Version)cited by 0
  24. 2026Evolving and Detecting Multi-Turn Deception using Geometric Signaturescited by 0
  25. 2026Towards High-Level Semantic Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  2. 2024Efficacy of Synthetic Data as a Benchmark
  3. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  4. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
  5. 2023Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models
  6. 2025Scaling Laws of Synthetic Data for Language Models
  7. 2025Artificial Conversations, Real Results: Fostering Language Detection with Synthetic Data
  8. 2024A Survey on Data Synthesis and Augmentation for Large Language Models