mlsanityData to 14 Oct 2026

SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains

arXiv 2507.07229 ↗First posted 9 Jul 2025cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

We present SynthTextEval, a toolkit for conducting comprehensive evaluations of synthetic text. The fluency of large language model (LLM) outputs has made synthetic text potentially viable for numerous applications, such as reducing the risks of privacy violations in the development and deployment of AI systems in high-stakes domains. Realizing this potential, however, requires principled consistent evaluations of synthetic data across multiple dimensions: its utility in downstream systems, the fairness of these systems, the risk of privacy leakage, general distributional differences from the source text, and qualitative feedback from domain experts. SynthTextEval allows users to conduct evaluations along all of these dimensions over synthetic data that they upload or generate using the toolkit's generation module. While our toolkit can be run over any data, we highlight its functionality and effectiveness over datasets from two high-stakes domains: healthcare and law. By consolidating and standardizing evaluation metrics, we aim to improve the viability of synthetic text, and in-turn, privacy-preservation in AI development.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 30 of its 54 references that are in the map. First posted on arXiv 9 Jul 2025.

Builds on

30 of 54 references.

References · 30

  1. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2018Hierarchical Neural Story Generationcited by 1,193
  5. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  6. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  7. 2022Large Language Models Can Self-Improvecited by 664
  8. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  9. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  10. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  11. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  12. 2021Numerical Composition of Differential Privacycited by 180
18 more
  1. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  2. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  3. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  4. 2022The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymizationcited by 95
  5. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 94
  6. 2023Synthcity: facilitating innovative use cases of synthetic data in different data modalitiescited by 76
  7. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  8. 2022A Customized Text Sanitization Mechanism with Differential Privacycited by 52
  9. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  10. 2024DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflowscited by 43
  11. 2023PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Modelscited by 37
  12. 2025A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakagecited by 16
  13. 2023Auditing and Generating Synthetic Data with Controllable Trust Trade-offscited by 13
  14. 2024Evaluating Differentially Private Synthetic Data Generation in High-Stakes Domainscited by 12
  15. 2023Assessment of Differentially Private Synthetic Data for Utility and Fairness in End-to-End Machine Learning Pipelines for Tabular Datacited by 8
  16. 2022Mention Annotations Alone Enable Efficient Domain Adaptation for Coreference Resolutioncited by 6
  17. 2024SafeSynthDP: Leveraging Large Language Models for Privacy-Preserving Synthetic Data Generation Using Differential Privacycited by 5
  18. 2024Privacy Evaluation Benchmarks for NLP Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025Controlled Generation for Private Synthetic Textcited by 2
  2. 2026NodeSynth: Socially Aligned Synthetic Data for AI Evaluationcited by 0
  3. 2026When Does Synthetic Patent Data Help? Volume-Fidelity Trade-offs in Low-Resource Multi-Label Classificationcited by 0
  4. 2026ROBE: Reversed-Order-Biased-Experts for Extracting Extreme Long-tail Events from Historical Textscited by 0
  5. 2026Schematize: An Agentic System for Generating and Refining Information-Extraction Schemas for Legal Researchcited by 0
  6. 2026Beyond Mode Collapse: Generating Diverse Synthetic Expert Conversations via Generative Flow Networkscited by 0
  7. 2026Synthetic Data Characterization via Training Dynamicscited by 0